# TensorPlay documentation URL: https://www.tensorplay.cn/docs/ TensorPlay documentation TensorPlay is a PyTorch-compatible tensor library for deep learning using GPUs and CPUs. Features described in this documentation are classified by release status: Stable (API-Stable): These features will be maintained long-term and there should generally be no major performance limitations or gaps in documentation. We also expect to maintain backwards compatibility (although breaking changes can happen and notice will be given one release ahead of time). Unstable (API-Unstable): Encompasses all features that are under active development where APIs may change based on user feedback, requisite performance improvements or because coverage across operators is not yet complete. The APIs and performance characteristics of these features may change. tensorplay Tensors Generators Random sampling Serialization Parallelism Locally disabling gradient computation Utilities Optimizations TensorPlay-specific additions Automatic differentiation package - tensorplay.autograd Automatic differentiation package - torch.autograd tensorplay._C._autograd.backward tensorplay.autograd.grad Functional higher level API Function Custom Function utilities Numerical gradient checking Debugging and anomaly detection tensorplay.nn Containers Convolution Layers Pooling layers Padding Layers Non-linear Activations (weighted sum, nonlinearity) Non-linear Activations (other) Normalization Layers Recurrent Layers Transformer Layers Linear Layers Dropout Layers Sparse Layers Distance Functions Loss Functions Vision Layers Shuffle Layers DataParallel Layers (multi-GPU, distributed) Utilities Lazy Modules Initialization TensorPlay-specific additions tensorplay.nn.functional Convolution functions Pooling functions Attention Mechanisms Non-linear activation functions Linear functions Dropout functions Sparse functions Distance functions Loss functions Vision functions Low-Precision functions TensorPlay-specific additions tensorplay.nn.init torch.nn.init tensorplay.nn.init.calculate_gain tensorplay.nn.init.uniform_ tensorplay.nn.init.normal_ tensorplay.nn.init.constant_ tensorplay.nn.init.ones_ tensorplay.nn.init.zeros_ tensorplay.nn.init.eye_ tensorplay.nn.init.dirac_ tensorplay.nn.init.xavier_uniform_ tensorplay.nn.init.xavier_normal_ tensorplay.nn.init.kaiming_uniform_ tensorplay.nn.init.kaiming_normal_ tensorplay.nn.init.trunc_normal_ tensorplay.nn.init.orthogonal_ tensorplay.nn.init.sparse_ tensorplay.optim Base class Module-level hooks Utilities Algorithms How to adjust learning rate tensorplay.cuda torch.cuda StreamContext tensorplay.cuda.can_device_access_peer tensorplay.cuda.check_error tensorplay.cuda.current_blas_handle tensorplay.cuda.current_solver_handle tensorplay.cuda.current_device tensorplay.cuda.current_stream tensorplay.cuda.cudart tensorplay.cuda.default_stream device tensorplay.cuda.device_count tensorplay.cuda.device_memory_used device_of tensorplay.cuda.get_arch_list tensorplay.cuda.get_device_capability tensorplay.cuda.get_device_name tensorplay.cuda.get_device_properties tensorplay.cuda.get_gencode_flags tensorplay.cuda.get_stream_from_external tensorplay.cuda.get_sync_debug_mode tensorplay.cuda.init tensorplay.cuda.ipc_collect tensorplay.cuda.is_available tensorplay.cuda.is_bf16_supported tensorplay.cuda.is_initialized tensorplay.cuda.is_tf32_supported tensorplay.cuda.memory_usage tensorplay.cuda.set_device tensorplay.cuda.set_stream tensorplay.cuda.set_sync_debug_mode tensorplay.cuda.stream tensorplay.cuda.synchronize tensorplay.cuda.utilization tensorplay.cuda.temperature tensorplay.cuda.power_draw tensorplay.cuda.clock_rate tensorplay.cuda.AcceleratorError tensorplay.cuda.OutOfMemoryError Random Number Generator Streams and events Graphs (beta) Graph Kernel Annotations (prototype) Memory management NVIDIA Tools Extension (NVTX) GPUDirect Storage (prototype) Green Contexts (experimental) TensorPlay-specific additions Automatic Mixed Precision package - tensorplay.amp Autocasting Gradient Scaling tensorplay.linalg Matrix Properties Decompositions Solvers Inverses Matrix Functions Matrix Products Tensor Operations Misc Experimental Functions tensorplay.fft Fast Fourier Transforms Helper Functions tensorplay.special Functions tensorplay.sparse Tensor methods and sparse tensorplay.random Random Generator tensorplay.utils.data Memory Pinning tensorplay.utils.checkpoint torch.utils.checkpoint tensorplay.utils.checkpoint.checkpoint tensorplay.utils.checkpoint.checkpoint_sequential CheckpointPolicy tensorplay.utils.checkpoint.set_checkpoint_early_stop tensorplay.futures torch.futures Future tensorplay.hub Loading models from Hub Multiprocessing package - tensorplay.multiprocessing API Reference tensorplay.library Choosing the kind of custom op Extending custom ops (created from Python or C++) Low-level APIs Distributed communication package - tensorplay.distributed Distributed communication package - torch.distributed tensorplay.distributed.rendezvous.register_rendezvous_handler tensorplay.distributed.algorithms.model_averaging.utils.average_parameters tensorplay.distributed.algorithms.model_averaging.utils.average_parameters_or_parameter_groups tensorplay.distributed.algorithms.model_averaging.utils.get_params_to_average Initialization Post-Initialization Groups DeviceMesh Point-to-point communication Collective functions Distributed Key-Value Store Launch utility Watchdog (Experimental) Quantization Quantization API Reference (Kept since APIs are still public) tensorplay.compiler Graph IR CUDA graphs Caching and errors Related modules tensorplay.stax Functions tensorplay.vision Top-level functions Submodules tensorplay.audio Top-level functions Submodules Indices and tables Index Module Index Search Page # Automatic Mixed Precision package - tensorplay.amp URL: https://www.tensorplay.cn/docs/amp.html Automatic Mixed Precision package - tensorplay.amp Autocasting tensorplay.amp.autocast_mode.is_autocast_available tensorplay.amp.autocast_mode.autocast Instances of autocast serve as context managers or decorators that allow regions of your script to run in mixed precision. tensorplay.amp.autocast_mode.custom_fwd Create a helper decorator for forward methods of custom autograd functions. tensorplay.amp.autocast_mode.custom_bwd Create a helper decorator for backward methods of custom autograd functions. Gradient Scaling tensorplay.amp.grad_scaler.GradScaler An instance scaler of GradScaler . tensorplay.amp.is_autocast_available # tensorplay.audio URL: https://www.tensorplay.cn/docs/audio.html tensorplay.audio tensorplay.audio is the audio I/O and processing toolkit of TensorPlay, mirroring the torchaudio package layout. The listing below is a static overview; backend availability depends on installed optional dependencies, so this page intentionally does not use autodoc. Top-level functions load , save , info — audio file I/O and metadata ( AudioMetaData ) check_available , list_audio_backends , get_audio_backend , set_audio_backend — backend discovery and selection Submodules tensorplay.audio.functional — functional audio operations tensorplay.audio.transforms — composable audio transforms tensorplay.audio.datasets — audio datasets ( CMUDict , …) tensorplay.audio.models — reference audio models tensorplay.audio.compliance — standards-compliance transforms tensorplay.audio.utils — shared helpers # Automatic differentiation package - tensorplay.autograd URL: https://www.tensorplay.cn/docs/autograd.html Automatic differentiation package - tensorplay.autograd Automatic differentiation package - torch.autograd tensorplay._C._autograd.backward tensorplay.autograd.grad Compute and return the sum of gradients of outputs with respect to the inputs. Functional higher level API tensorplay.autograd.functional.jacobian Compute the Jacobian of a given function. tensorplay.autograd.functional.hessian Compute the Hessian of a given scalar function. tensorplay.autograd.functional.vjp Compute the dot product between a vector v and the Jacobian of the given function at the point given by the inputs. tensorplay.autograd.functional.jvp Compute the dot product between the Jacobian of the given function at the point given by the inputs and a vector v . tensorplay.autograd.functional.vhp Compute the dot product between vector v and Hessian of a given scalar function at a specified point. tensorplay.autograd.functional.hvp Compute the dot product between the scalar function's Hessian and a vector v at a specified point. Function tensorplay.autograd.function.Function Records operation history and defines formulas for differentiating ops. tensorplay.autograd.backward tensorplay.autograd.jvp Compute the dot product between the Jacobian of the given function at the point given by the inputs and a vector v . Custom Function utilities tensorplay.autograd.function.once_differentiable Decorator to make a custom autograd Function's backward run once, with gradients detached and grad-mode disabled inside. Numerical gradient checking tensorplay.autograd.gradcheck.gradcheck Check gradients computed via small finite differences against analytical gradients wrt tensors in inputs that are of floating point or complex type and with requires_grad=True . tensorplay.autograd.gradcheck.gradgradcheck Check gradients of gradients computed via small finite differences against analytical gradients wrt tensors in inputs and grad_outputs that are of floating point or complex type and with requires_grad=True . tensorplay.autograd.gradcheck.GradcheckError Error raised by gradcheck() and gradgradcheck() . tensorplay.autograd.gradcheck.get_analytical_jacobian tensorplay.autograd.gradcheck.get_numerical_jacobian Compute the numerical Jacobian for a given fn and its inputs. tensorplay.autograd.gradcheck.get_numerical_jacobian_wrt_specific_input Debugging and anomaly detection tensorplay.autograd.anomaly_mode.detect_anomaly Context-manager that enables anomaly detection for the autograd engine. tensorplay.autograd.anomaly_mode.set_detect_anomaly Context-manager that sets the anomaly detection for the autograd engine on or off. # tensorplay.utils.checkpoint URL: https://www.tensorplay.cn/docs/checkpoint.html tensorplay.utils.checkpoint torch.utils.checkpoint tensorplay.utils.checkpoint.checkpoint Runs function(*args, **kwargs) eagerly (no recomputation). tensorplay.utils.checkpoint.checkpoint_sequential torch.utils.checkpoint.checkpoint_sequential compatibility. tensorplay.utils.checkpoint.CheckpointPolicy tensorplay.utils.checkpoint.set_checkpoint_early_stop No-op for API parity. # tensorplay.compiler URL: https://www.tensorplay.cn/docs/compiler.html tensorplay.compiler TensorPlay ships an in-process graph compiler used by tensorplay.stax and the CUDA graphs workflow. This module is TensorPlay-specific and has no direct upstream torch counterpart (it is conceptually closest to torch.compile internals plus torch.cuda.graphs ). The reference below is a static listing: the underlying extension modules are under active development, so this page intentionally does not use autodoc. Graph IR Graph , Node , GraphModule — graph representation and module wrapper PassBase , DecomposePass , ConstFold , DeadCodeElimination , NormalizeOperators — compiler passes NodePathTracer — attribution helper CUDA graphs CudaGraphManager , CudaGraphError — capture/replay management Guard , GuardChain — shape/guard caching for replay validation Caching and errors CodeCache — compiled artifact cache AOTError , GraphCaptureError — error types Related modules tensorplay.jit — tracing/scripting entry points ( script , trace , export , ignore , is_tracing ) tensorplay.export — ExportedProgram / GraphSignature export format tensorplay.cuda.CUDAGraph — low-level stream capture API # tensorplay.cuda URL: https://www.tensorplay.cn/docs/cuda.html tensorplay.cuda torch.cuda tensorplay.cuda.StreamContext Context-manager that selects a given stream. tensorplay.cuda.can_device_access_peer Check if peer access between two devices is possible. tensorplay.cuda.check_error Raise an error if the result of a CUDA runtime API call is not success. tensorplay.cuda.current_blas_handle Return cublasHandle_t pointer to current cuBLAS handle tensorplay.cuda.current_solver_handle Return cusolverDnHandle_t pointer to current cuSOLVER handle tensorplay.cuda.current_device Return the index of a currently selected device. tensorplay.cuda.current_stream Return the currently selected Stream for a given device. tensorplay.cuda.cudart Retrieves the CUDA runtime API module. tensorplay.cuda.default_stream Return the default Stream for a given device. tensorplay.cuda.device Context-manager that changes the selected device. tensorplay.cuda.device_count Return the number of GPUs available. tensorplay.cuda.device_memory_used Return used global (device) memory in bytes as given by nvidia-smi . tensorplay.cuda.device_of Context-manager that changes the current device to that of given object. tensorplay.cuda.get_arch_list Return list CUDA architectures this library was compiled for. tensorplay.cuda.get_device_capability Get the cuda capability of a device. tensorplay.cuda.get_device_name Get the name of a device. tensorplay.cuda.get_device_properties Get the properties of a device. tensorplay.cuda.get_gencode_flags Return NVCC gencode flags this library was compiled with. tensorplay.cuda.get_stream_from_external Return a Stream from an externally allocated CUDA stream. tensorplay.cuda.get_sync_debug_mode Return current value of debug mode for cuda synchronizing operations. tensorplay.cuda.init Initialize TensorPlay's CUDA state. tensorplay.cuda.ipc_collect Force collects GPU memory after it has been released by CUDA IPC. tensorplay.cuda.is_available Return a bool indicating if CUDA is currently available. tensorplay.cuda.is_bf16_supported Return a bool indicating if the current CUDA device supports dtype bfloat16. tensorplay.cuda.is_initialized Return whether TensorPlay's CUDA state has been initialized. tensorplay.cuda.is_tf32_supported Return a bool indicating if the current CUDA device supports dtype tf32. tensorplay.cuda.memory_usage Return the percent of time over the past sample period during which global (device) memory was being read or written as given by nvidia-smi . tensorplay.cuda.set_device Set the current device. tensorplay.cuda.set_stream Set the current stream. This is a wrapper API to set the stream. tensorplay.cuda.set_sync_debug_mode Set the debug mode for cuda synchronizing operations. tensorplay.cuda.stream Wrap around the Context-manager StreamContext that selects a given stream. tensorplay.cuda.synchronize Wait for all kernels in all streams on a CUDA device to complete. tensorplay.cuda.utilization Return the percent of time over the past sample period during which one or more kernels was executing on the GPU as given by nvidia-smi . tensorplay.cuda.temperature Return the average temperature of the GPU sensor in Degrees C (Centigrades). tensorplay.cuda.power_draw Return the average power draw of the GPU sensor in mW (MilliWatts) over the past sample period as given by nvidia-smi for Fermi or newer fully supported devices. tensorplay.cuda.clock_rate Return the clock speed of the GPU SM in MHz (megahertz) over the past sample period as given by nvidia-smi . tensorplay.cuda.AcceleratorError tensorplay.cuda.OutOfMemoryError Random Number Generator tensorplay.cuda.random.get_rng_state Return the random number generator state of the specified GPU as a ByteTensor. tensorplay.cuda.random.get_rng_state_all Return a list of ByteTensor representing the random number states of all devices. tensorplay.cuda.random.set_rng_state Set the random number generator state of the specified GPU. tensorplay.cuda.random.set_rng_state_all Set the random number generator state of all devices. tensorplay.cuda.random.manual_seed Set the seed for generating random numbers for the current GPU. tensorplay.cuda.random.manual_seed_all Set the seed for generating random numbers on all GPUs. tensorplay.cuda.random.seed Set the seed for generating random numbers to a random number for the current GPU. tensorplay.cuda.random.seed_all Set the seed for generating random numbers to a random number on all GPUs. tensorplay.cuda.random.initial_seed Return the current random seed of the current GPU. Streams and events tensorplay.cuda.streams.Stream Wrapper around a CUDA stream. tensorplay.cuda.streams.ExternalStream Wrapper around an externally allocated CUDA stream. tensorplay.cuda.streams.Event Wrapper around a CUDA event. Graphs (beta) tensorplay.cuda.graphs.is_current_stream_capturing Return True if CUDA graph capture is underway on the current thread. tensorplay.cuda.graphs.graph_pool_handle Return an opaque token representing the id of a graph memory pool. tensorplay.cuda.graphs.CUDAGraph Wrapper around a CUDA graph, mirroring torch.cuda.CUDAGraph . tensorplay.cuda.graphs.graph Context-manager that captures CUDA work into a tensorplay.cuda.CUDAGraph . tensorplay.cuda.graphs.make_graphed_callables Callables that run per-iteration with CUDA graph capture (not supported). tensorplay.cuda.graphs.export_dot Export the last captured CUDA graph to a DOT file. tensorplay.cuda.graphs.export_graph_data Serialize a captured CUDA graph into a dictionary of node data. Graph Kernel Annotations (prototype) tensorplay.cuda.graph_annotations.is_available Whether graph-kernel annotation recording is supported (always False here). tensorplay.cuda.graph_annotations.mark_kernels Tag the GPU work captured within its scope with user metadata. tensorplay.cuda.graph_annotations.get_kernel_annotations Return the recorded kernel annotations mapping. tensorplay.cuda.graph_annotations.clear_kernel_annotations Clear all recorded kernel annotations. Memory management tensorplay.cuda.memory.caching_allocator_disabled Context manager that temporarily disables the CUDA caching allocator. tensorplay.cuda.memory.caching_allocator_enable Enable or disable the CUDA memory allocator. tensorplay.cuda.memory.use_mem_pool Route allocations to a MemPool (not supported by this build). tensorplay.cuda.nccl.version Returns the version of the NCCL. tensorplay.cuda.profiler.profile Enable profiling. tensorplay.cuda.profiler.start Starts cuda profiler data collection. tensorplay.cuda.profiler.stop Stops cuda profiler data collection. NVIDIA Tools Extension (NVTX) tensorplay.cuda.nvtx.mark Describe an instantaneous event that occurred at some point. tensorplay.cuda.nvtx.range_push Push a range onto a stack of nested range span. tensorplay.cuda.nvtx.range_pop Pop a range off of a stack of nested range spans. tensorplay.cuda.nvtx.range Context manager / decorator that pushes an NVTX range at the beginning of its scope, and pops it at the end. tensorplay.cuda.nvtx.range_end Mark the end of a range for a given range_id. tensorplay.cuda.nvtx.range_start Mark the start of a range with string message. GPUDirect Storage (prototype) tensorplay.cuda.gds.GdsFile A GDS file handle (not supported by this TensorPlay build). Green Contexts (experimental) tensorplay.cuda.green_contexts.GreenContext tensorplay.cuda.nccl.is_available TensorPlay-specific additions CUDAPluggableAllocator CUDA memory allocator plugin (not supported by this TensorPlay build). CudaError DeferredCudaCallError Device MemPool MemPool context (not supported by this TensorPlay build). caching_allocator_alloc Perform a memory allocation using the CUDA memory allocator. caching_allocator_delete Delete memory allocated using the CUDA memory allocator. change_current_allocator Change the currently used memory allocator (not supported by this build). classproperty empty_cache Release all unoccupied cached memory currently held by the caching allocator so that those can be used in other GPU application and visible in nvidia-smi . get_allocator_backend Returns the active allocator backend as a string. get_per_process_memory_fraction Get memory fraction for a process. host_memory_stats Return a dictionary of pinned (host) allocator statistics. host_memory_stats_as_nested_dict Return the result of host_memory_stats() as a nested dictionary. is_gds_available Return whether GDS is available. list_gpu_processes Return a human-readable printout of the running processes and their GPU memory use for a given device. max_memory_allocated Return the maximum GPU memory occupied by tensors in bytes for a given device. max_memory_cached Deprecated; see max_memory_reserved() . max_memory_reserved Return the maximum GPU memory managed by the caching allocator in bytes for a given device. mem_get_info Return the global free and total GPU memory for a given device using cudaMemGetInfo. memory_allocated Return the current GPU memory occupied by tensors in bytes for a given device. memory_cached Deprecated; see memory_reserved() . memory_reserved Return the current GPU memory managed by the caching allocator in bytes for a given device. memory_snapshot Return a snapshot of the CUDA memory allocator state across all devices. memory_stats Return a dictionary of CUDA memory allocator statistics for a given device. memory_stats_as_nested_dict Return the result of memory_stats() as a nested dictionary. memory_summary Return a human-readable printout of the current memory allocator statistics for a given device. reset_accumulated_host_memory_stats Reset the "accumulated" (historical) stats tracked by the host memory allocator. reset_accumulated_memory_stats Reset the "accumulated" (historical) stats tracked by the CUDA memory allocator. reset_max_memory_allocated Reset the starting point in tracking maximum GPU memory occupied by tensors for a given device. reset_max_memory_cached Reset the starting point in tracking maximum GPU memory managed by the caching allocator for a given device. reset_peak_host_memory_stats Reset the "peak" stats tracked by the host memory allocator. reset_peak_memory_stats Reset the "peak" stats tracked by the CUDA memory allocator. set_per_process_memory_fraction Set memory fraction for a process. # tensorplay.utils.data URL: https://www.tensorplay.cn/docs/data.html tensorplay.utils.data Memory Pinning tensorplay.utils.data.DataLoader Data loader combines a dataset and a sampler, and provides an iterable over the given dataset. tensorplay.utils.data.Dataset An abstract class representing a Dataset . tensorplay.utils.data.IterableDataset An iterable Dataset. tensorplay.utils.data.TensorDataset Dataset wrapping tensors. tensorplay.utils.data.StackDataset Dataset as a stacking of multiple datasets. tensorplay.utils.data.ConcatDataset Dataset as a concatenation of multiple datasets. tensorplay.utils.data.ChainDataset Dataset for chaining multiple IterableDataset s. tensorplay.utils.data.Subset Subset of a dataset at specified indices. tensorplay.utils.data.default_collate Take in a batch of data and put the elements within the batch into a tensor with an additional outer dimension - batch size. tensorplay.utils.data.default_convert Convert each NumPy array element into a tensorplay.Tensor . tensorplay.utils.data.get_worker_info Returns the information about the current DataLoader iterator worker process. tensorplay.utils.data.Sampler Base class for all Samplers. tensorplay.utils.data.SequentialSampler Samples elements sequentially, always in the same order. tensorplay.utils.data.RandomSampler Samples elements randomly. tensorplay.utils.data.SubsetRandomSampler Samples elements randomly from a given list of indices, without replacement. tensorplay.utils.data.WeightedRandomSampler Samples elements from [0,..,len(weights)-1] with given probabilities (weights). tensorplay.utils.data.BatchSampler Wraps another sampler to yield a mini-batch of indices. # Distributed communication package - tensorplay.distributed URL: https://www.tensorplay.cn/docs/distributed.html Distributed communication package - tensorplay.distributed Distributed communication package - torch.distributed tensorplay.distributed.rendezvous.register_rendezvous_handler Register a new rendezvous handler. tensorplay.distributed.algorithms.model_averaging.utils.average_parameters Averages all the given parameters. tensorplay.distributed.algorithms.model_averaging.utils.average_parameters_or_parameter_groups Averages parameters of a model or parameter groups of an optimizer. tensorplay.distributed.algorithms.model_averaging.utils.get_params_to_average Return a list of parameters that need to average. Initialization tensorplay.distributed.distributed_c10d.is_available tensorplay.distributed.distributed_c10d.init_process_group tensorplay.distributed.device_mesh.init_device_mesh Initializes a DeviceMesh based on device_type and mesh_shape (torch parity). tensorplay.distributed.distributed_c10d.is_initialized tensorplay.distributed.distributed_c10d.is_mpi_available tensorplay.distributed.distributed_c10d.is_nccl_available tensorplay.distributed.distributed_c10d.is_gloo_available tensorplay.distributed.distributed_c10d.batch_isend_irecv Send or receive a batch of tensors asynchronously (torch parity). tensorplay.distributed.distributed_c10d.destroy_process_group tensorplay.distributed.distributed_c10d.irecv Receives a tensor asynchronously (torch parity; src global rank). Post-Initialization tensorplay.distributed.distributed_c10d.Backend tensorplay.distributed.distributed_c10d.get_backend tensorplay.distributed.distributed_c10d.get_rank tensorplay.distributed.distributed_c10d.get_world_size Groups tensorplay.distributed.distributed_c10d.new_group tensorplay.distributed.distributed_c10d.get_group_rank Translate a global rank into a group rank (torch parity). tensorplay.distributed.distributed_c10d.get_global_rank Translate a group rank into a global rank (torch parity). tensorplay.distributed.distributed_c10d.get_process_group_ranks Get all ranks associated with group (torch parity). DeviceMesh tensorplay.distributed.device_mesh.DeviceMesh DeviceMesh represents a mesh of devices (torch parity). Point-to-point communication tensorplay.distributed.distributed_c10d.send Send a tensor synchronously (torch parity). tensorplay.distributed.distributed_c10d.recv Receives a tensor synchronously; returns the sender rank. tensorplay.distributed.distributed_c10d.isend Send a tensor asynchronously (torch parity; dst is a global rank). tensorplay.distributed.distributed_c10d.send_object_list Sends picklable objects in object_list synchronously (torch parity). tensorplay.distributed.distributed_c10d.recv_object_list Receives picklable objects in object_list synchronously (torch parity). tensorplay.distributed.distributed_c10d.P2POp A class to build point-to-point operations for batch_isend_irecv . Collective functions tensorplay.distributed.distributed_c10d.broadcast tensorplay.distributed.distributed_c10d.broadcast_object_list Broadcasts picklable objects in object_list to the whole group. tensorplay.distributed.distributed_c10d.all_reduce tensorplay.distributed.distributed_c10d.reduce tensorplay.distributed.distributed_c10d.all_gather tensorplay.distributed.distributed_c10d.all_gather_object Gathers picklable objects from the whole group into a list (torch parity). tensorplay.distributed.distributed_c10d.gather tensorplay.distributed.distributed_c10d.gather_object Gathers picklable objects from the whole group in a single process. tensorplay.distributed.distributed_c10d.scatter tensorplay.distributed.distributed_c10d.scatter_object_list Scatters picklable objects to the whole group (torch parity). tensorplay.distributed.distributed_c10d.reduce_scatter tensorplay.distributed.distributed_c10d.all_to_all_single Splits input evenly (or by split sizes) and scatters the chunks. tensorplay.distributed.distributed_c10d.all_to_all Scatters a list of tensors to ranks and collects one from each. tensorplay.distributed.distributed_c10d.barrier tensorplay.distributed.distributed_c10d.Work Handle for an async collective (torch.distributed.Work subset). tensorplay.distributed.distributed_c10d.ReduceOp Distributed Key-Value Store tensorplay.distributed._store.Store tensorplay.distributed._store.TCPStore Client for a threaded TCP key-value server ( torch.distributed.TCPStore subset: set/get/add/delete/check as used by rendezvous and barriers). tensorplay.distributed._store.FileStore Flock-based append-log store in a single file. Launch utility tensorplay.distributed.launch.launch tensorplay.distributed.launch.main tensorplay.distributed.launch.parse_args Watchdog (Experimental) tensorplay.distributed.collective_utils.all_gather_object_enforce_type Similar to plain all_gather_object but with additional type checking AFTER gather is done to ensure basic consistency. # tensorplay.fft URL: https://www.tensorplay.cn/docs/fft.html tensorplay.fft Fast Fourier Transforms tensorplay.fft.fft Computes the one-dimensional discrete Fourier transform. tensorplay.fft.ifft Computes the one-dimensional inverse discrete Fourier transform. tensorplay.fft.fft2 Computes the two-dimensional discrete Fourier transform. tensorplay.fft.ifft2 Computes the two-dimensional inverse discrete Fourier transform. tensorplay.fft.fftn Computes the N-dimensional discrete Fourier transform over dim . tensorplay.fft.ifftn Computes the N-dimensional inverse discrete Fourier transform. tensorplay.fft.rfft Computes the one-dimensional FFT of real input, one-sided output. tensorplay.fft.irfft Computes the inverse of rfft() ; n is the output length. tensorplay.fft.rfft2 Two-dimensional FFT of real input: fft() on leading dims, then rfft() on the final transformed dimension. tensorplay.fft.irfft2 Inverse of rfft2() : irfft() on the final dimension first ( s[-1] is the real output size), then ifft() on the rest. tensorplay.fft.rfftn N-dimensional FFT of real input; one-sided along the last listed dim. tensorplay.fft.irfftn Inverse of rfftn() ; s[-1] is the real output size. tensorplay.fft.hfft Computes the 1-D FFT of a Hermitian-symmetric spectrum; real output. tensorplay.fft.ihfft Computes the inverse of hfft() ; one-sided complex output. tensorplay.fft.hfft2 Two-dimensional inverse of a Hermitian-symmetric spectrum; real output. tensorplay.fft.ihfft2 Two-dimensional counterpart of ihfft() . tensorplay.fft.hfftn N-dimensional FFT of a Hermitian-symmetric spectrum; real output. tensorplay.fft.ihfftn Inverse of hfftn() : ihfft() along the final transformed dimension, then fft() over the remaining dimensions. Helper Functions tensorplay.fft.fftfreq DFT sample frequencies (cycles/unit): [0, 1, ..., n/2-1, -n/2, ..., -1] / (n*d) . tensorplay.fft.rfftfreq Sample frequencies for rfft() /one-sided transforms: [0..n//2] / (n*d) . tensorplay.fft.fftshift Re-orders an N-D FFT output so the zero-frequency term is centered. tensorplay.fft.ifftshift Inverse of fftshift() ; shifts by -(n // 2) (odd-safe). # tensorplay.futures URL: https://www.tensorplay.cn/docs/futures.html tensorplay.futures torch.futures tensorplay.futures.Future Holder for an asynchronous result (torch parity subset). # tensorplay._C._autograd.backward URL: https://www.tensorplay.cn/docs/generated/tensorplay._C._autograd.backward.html tensorplay._C._autograd.backward tensorplay._C._autograd. backward ( tensors : collections.abc.Sequence [ tensorplay._C.TensorBase ] , grad_tensors : collections.abc.Sequence [ tensorplay._C.TensorBase ] | None = None , retain_graph : bool | None = None , create_graph : bool = False ) → None # tensorplay._composite_funcs.absolute URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.absolute.html tensorplay._composite_funcs.absolute tensorplay._composite_funcs. absolute ( input ) [source] Alias of abs() . # tensorplay._composite_funcs.acos_ URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.acos_.html tensorplay._composite_funcs.acos_ tensorplay._composite_funcs. acos_ ( self ) # tensorplay._composite_funcs.acosh_ URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.acosh_.html tensorplay._composite_funcs.acosh_ tensorplay._composite_funcs. acosh_ ( self ) # tensorplay._composite_funcs.adaptive_avg_pool1d URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.adaptive_avg_pool1d.html tensorplay._composite_funcs.adaptive_avg_pool1d tensorplay._composite_funcs. adaptive_avg_pool1d ( input , output_size ) [source] # tensorplay._composite_funcs.adaptive_max_pool1d URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.adaptive_max_pool1d.html tensorplay._composite_funcs.adaptive_max_pool1d tensorplay._composite_funcs. adaptive_max_pool1d ( input , output_size ) [source] # tensorplay._composite_funcs.adjoint URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.adjoint.html tensorplay._composite_funcs.adjoint tensorplay._composite_funcs. adjoint ( input ) [source] # tensorplay._composite_funcs.alias_copy URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.alias_copy.html tensorplay._composite_funcs.alias_copy tensorplay._composite_funcs. alias_copy ( input ) [source] # tensorplay._composite_funcs.arccos URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.arccos.html tensorplay._composite_funcs.arccos tensorplay._composite_funcs. arccos ( input ) [source] # tensorplay._composite_funcs.arccosh URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.arccosh.html tensorplay._composite_funcs.arccosh tensorplay._composite_funcs. arccosh ( input ) [source] # tensorplay._composite_funcs.arcsin URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.arcsin.html tensorplay._composite_funcs.arcsin tensorplay._composite_funcs. arcsin ( input ) [source] # tensorplay._composite_funcs.arcsinh URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.arcsinh.html tensorplay._composite_funcs.arcsinh tensorplay._composite_funcs. arcsinh ( input ) [source] # tensorplay._composite_funcs.arctan URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.arctan.html tensorplay._composite_funcs.arctan tensorplay._composite_funcs. arctan ( input ) [source] # tensorplay._composite_funcs.arctan2 URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.arctan2.html tensorplay._composite_funcs.arctan2 tensorplay._composite_funcs. arctan2 ( input , other ) [source] atan2(y, x) = 2*atan(y / (hypot + x)); native atan2 CPU kernel is currently unregistered in this tree, so compose. Edge narrows vs torch: negative-zero y with x < 0 returns +pi instead of -pi. # tensorplay._composite_funcs.arctanh URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.arctanh.html tensorplay._composite_funcs.arctanh tensorplay._composite_funcs. arctanh ( input ) [source] # tensorplay._composite_funcs.asin_ URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.asin_.html tensorplay._composite_funcs.asin_ tensorplay._composite_funcs. asin_ ( self ) # tensorplay._composite_funcs.asinh_ URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.asinh_.html tensorplay._composite_funcs.asinh_ tensorplay._composite_funcs. asinh_ ( self ) # tensorplay._composite_funcs.atan_ URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.atan_.html tensorplay._composite_funcs.atan_ tensorplay._composite_funcs. atan_ ( self ) # tensorplay._composite_funcs.atanh_ URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.atanh_.html tensorplay._composite_funcs.atanh_ tensorplay._composite_funcs. atanh_ ( self ) # tensorplay._composite_funcs.avg_pool1d URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.avg_pool1d.html tensorplay._composite_funcs.avg_pool1d tensorplay._composite_funcs. avg_pool1d ( input , kernel_size , stride = None , padding = 0 , ceil_mode = False , count_include_pad = True ) [source] # tensorplay._composite_funcs.can_cast URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.can_cast.html tensorplay._composite_funcs.can_cast tensorplay._composite_funcs. can_cast ( from_ , to ) [source] # tensorplay._composite_funcs.cartesian_prod URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.cartesian_prod.html tensorplay._composite_funcs.cartesian_prod tensorplay._composite_funcs. cartesian_prod ( * tensors ) [source] # tensorplay._composite_funcs.chain_matmul URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.chain_matmul.html tensorplay._composite_funcs.chain_matmul tensorplay._composite_funcs. chain_matmul ( * matrices ) [source] # tensorplay._composite_funcs.combinations URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.combinations.html tensorplay._composite_funcs.combinations tensorplay._composite_funcs. combinations ( input , r = 2 , with_replacement = False ) [source] # tensorplay._composite_funcs.concat URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.concat.html tensorplay._composite_funcs.concat tensorplay._composite_funcs. concat ( tensors , dim = 0 , * , out = None ) [source] # tensorplay._composite_funcs.concatenate URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.concatenate.html tensorplay._composite_funcs.concatenate tensorplay._composite_funcs. concatenate ( tensors , dim = 0 , * , out = None ) [source] # tensorplay._composite_funcs.copysign URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.copysign.html tensorplay._composite_funcs.copysign tensorplay._composite_funcs. copysign ( input , other ) [source] # tensorplay._composite_funcs.corrcoef URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.corrcoef.html tensorplay._composite_funcs.corrcoef tensorplay._composite_funcs. corrcoef ( input ) [source] # tensorplay._composite_funcs.cosine_similarity URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.cosine_similarity.html tensorplay._composite_funcs.cosine_similarity tensorplay._composite_funcs. cosine_similarity ( x1 , x2 , dim = 1 , eps = 1e-08 ) [source] # tensorplay._composite_funcs.cov URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.cov.html tensorplay._composite_funcs.cov tensorplay._composite_funcs. cov ( input , * , correction = 1 , fweights = None , aweights = None ) [source] Each row of input is a variable, each column an observation (torch semantics; no rowvar argument). # tensorplay._composite_funcs.cumulative_trapezoid URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.cumulative_trapezoid.html tensorplay._composite_funcs.cumulative_trapezoid tensorplay._composite_funcs. cumulative_trapezoid ( y , x = None , * , dx = None , dim = -1 ) [source] # tensorplay._composite_funcs.detach URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.detach.html tensorplay._composite_funcs.detach tensorplay._composite_funcs. detach ( input ) [source] # tensorplay._composite_funcs.diagflat URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.diagflat.html tensorplay._composite_funcs.diagflat tensorplay._composite_funcs. diagflat ( input , offset = 0 ) [source] # tensorplay._composite_funcs.diagonal_copy URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.diagonal_copy.html tensorplay._composite_funcs.diagonal_copy tensorplay._composite_funcs. diagonal_copy ( input , diagonal = 0 , dim1 = -2 , dim2 = -1 ) [source] # tensorplay._composite_funcs.divide URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.divide.html tensorplay._composite_funcs.divide tensorplay._composite_funcs. divide ( input , other , * , rounding_mode = None , out = None ) [source] # tensorplay._composite_funcs.expand_copy URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.expand_copy.html tensorplay._composite_funcs.expand_copy tensorplay._composite_funcs. expand_copy ( input , size , * , implicit = False ) [source] # tensorplay._composite_funcs.floor_divide URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.floor_divide.html tensorplay._composite_funcs.floor_divide tensorplay._composite_funcs. floor_divide ( input , other , * , out = None ) [source] # tensorplay._composite_funcs.fmod URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.fmod.html tensorplay._composite_funcs.fmod tensorplay._composite_funcs. fmod ( input , other , * , out = None ) [source] # tensorplay._composite_funcs.ger URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.ger.html tensorplay._composite_funcs.ger tensorplay._composite_funcs. ger ( input , vec2 ) [source] # tensorplay._composite_funcs.get_device URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.get_device.html tensorplay._composite_funcs.get_device tensorplay._composite_funcs. get_device ( input ) [source] # tensorplay._composite_funcs.gradient URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.gradient.html tensorplay._composite_funcs.gradient tensorplay._composite_funcs. gradient ( input , * , spacing = None , dim = None , edge_order = 1 ) [source] # tensorplay._composite_funcs.histc URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.histc.html tensorplay._composite_funcs.histc tensorplay._composite_funcs. histc ( input , bins = 100 , min = 0 , max = 0 ) [source] # tensorplay._composite_funcs.histogram URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.histogram.html tensorplay._composite_funcs.histogram tensorplay._composite_funcs. histogram ( input , bins = 10 , range = None , * , weight = None , density = False ) [source] # tensorplay._composite_funcs.is_conj URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.is_conj.html tensorplay._composite_funcs.is_conj tensorplay._composite_funcs. is_conj ( input ) [source] # tensorplay._composite_funcs.is_neg URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.is_neg.html tensorplay._composite_funcs.is_neg tensorplay._composite_funcs. is_neg ( input ) [source] # tensorplay._composite_funcs.is_nonzero URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.is_nonzero.html tensorplay._composite_funcs.is_nonzero tensorplay._composite_funcs. is_nonzero ( input ) [source] # tensorplay._composite_funcs.is_same_size URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.is_same_size.html tensorplay._composite_funcs.is_same_size tensorplay._composite_funcs. is_same_size ( input , other ) [source] # tensorplay._composite_funcs.isin URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.isin.html tensorplay._composite_funcs.isin tensorplay._composite_funcs. isin ( elements , test_elements , * , assume_unique = False , invert = False ) [source] # tensorplay._composite_funcs.kaiser_window URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.kaiser_window.html tensorplay._composite_funcs.kaiser_window tensorplay._composite_funcs. kaiser_window ( window_length , periodic = True , beta = 12.0 , * , dtype = None , layout = None , requires_grad = False ) [source] # tensorplay._composite_funcs.kron URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.kron.html tensorplay._composite_funcs.kron tensorplay._composite_funcs. kron ( input , other ) [source] # tensorplay._composite_funcs.lstm_cell URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.lstm_cell.html tensorplay._composite_funcs.lstm_cell tensorplay._composite_funcs. lstm_cell ( input , hx , cx , w_ih , w_hh , b_ih = None , b_hh = None ) [source] # tensorplay._composite_funcs.matrix_power URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.matrix_power.html tensorplay._composite_funcs.matrix_power tensorplay._composite_funcs. matrix_power ( input , n ) [source] # tensorplay._composite_funcs.max_pool1d URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.max_pool1d.html tensorplay._composite_funcs.max_pool1d tensorplay._composite_funcs. max_pool1d ( input , kernel_size , stride = None , padding = 0 , dilation = 1 , ceil_mode = False ) [source] # tensorplay._composite_funcs.multiply URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.multiply.html tensorplay._composite_funcs.multiply tensorplay._composite_funcs. multiply ( input , other , * , out = None ) [source] # tensorplay._composite_funcs.nanquantile URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.nanquantile.html tensorplay._composite_funcs.nanquantile tensorplay._composite_funcs. nanquantile ( input , q , dim = None , keepdim = False , * , interpolation = 'linear' ) [source] # tensorplay._composite_funcs.narrow_copy URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.narrow_copy.html tensorplay._composite_funcs.narrow_copy tensorplay._composite_funcs. narrow_copy ( input , dim , start , length ) [source] # tensorplay._composite_funcs.numel URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.numel.html tensorplay._composite_funcs.numel tensorplay._composite_funcs. numel ( obj ) [source] # tensorplay._composite_funcs.permute_copy URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.permute_copy.html tensorplay._composite_funcs.permute_copy tensorplay._composite_funcs. permute_copy ( input , dims ) [source] # tensorplay._composite_funcs.promote_types URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.promote_types.html tensorplay._composite_funcs.promote_types tensorplay._composite_funcs. promote_types ( type1 , type2 ) [source] # tensorplay._composite_funcs.put URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.put.html tensorplay._composite_funcs.put tensorplay._composite_funcs. put ( input , index , source ) [source] # tensorplay._composite_funcs.quantile URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.quantile.html tensorplay._composite_funcs.quantile tensorplay._composite_funcs. quantile ( input , q , dim = None , keepdim = False , * , interpolation = 'linear' ) [source] # tensorplay._composite_funcs.remainder URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.remainder.html tensorplay._composite_funcs.remainder tensorplay._composite_funcs. remainder ( input , other , * , out = None ) [source] # tensorplay._composite_funcs.repeat_interleave URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.repeat_interleave.html tensorplay._composite_funcs.repeat_interleave tensorplay._composite_funcs. repeat_interleave ( input , repeats , dim = None , * , output_size = None ) [source] # tensorplay._composite_funcs.resolve_conj URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.resolve_conj.html tensorplay._composite_funcs.resolve_conj tensorplay._composite_funcs. resolve_conj ( input ) [source] # tensorplay._composite_funcs.resolve_neg URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.resolve_neg.html tensorplay._composite_funcs.resolve_neg tensorplay._composite_funcs. resolve_neg ( input ) [source] # tensorplay._composite_funcs.result_type URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.result_type.html tensorplay._composite_funcs.result_type tensorplay._composite_funcs. result_type ( * args ) [source] # tensorplay._composite_funcs.rms_norm URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.rms_norm.html tensorplay._composite_funcs.rms_norm tensorplay._composite_funcs. rms_norm ( input , normalized_shape , weight = None , eps = None ) [source] # tensorplay._composite_funcs.rnn_relu_cell URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.rnn_relu_cell.html tensorplay._composite_funcs.rnn_relu_cell tensorplay._composite_funcs. rnn_relu_cell ( input , hx , w_ih , w_hh , b_ih = None , b_hh = None ) [source] # tensorplay._composite_funcs.rnn_tanh_cell URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.rnn_tanh_cell.html tensorplay._composite_funcs.rnn_tanh_cell tensorplay._composite_funcs. rnn_tanh_cell ( input , hx , w_ih , w_hh , b_ih = None , b_hh = None ) [source] # tensorplay._composite_funcs.rsub URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.rsub.html tensorplay._composite_funcs.rsub tensorplay._composite_funcs. rsub ( input , other , * , alpha = 1 ) [source] # tensorplay._composite_funcs.scalar_tensor URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.scalar_tensor.html tensorplay._composite_funcs.scalar_tensor tensorplay._composite_funcs. scalar_tensor ( s , * , dtype = None , device = None ) [source] # tensorplay._composite_funcs.select_copy URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.select_copy.html tensorplay._composite_funcs.select_copy tensorplay._composite_funcs. select_copy ( input , dim , index ) [source] # tensorplay._composite_funcs.slice_copy URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.slice_copy.html tensorplay._composite_funcs.slice_copy tensorplay._composite_funcs. slice_copy ( input , dim = 0 , start = None , end = None , step = 1 ) [source] # tensorplay._composite_funcs.split_copy URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.split_copy.html tensorplay._composite_funcs.split_copy tensorplay._composite_funcs. split_copy ( input , split_size_or_sections , dim = 0 ) [source] # tensorplay._composite_funcs.squeeze_copy URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.squeeze_copy.html tensorplay._composite_funcs.squeeze_copy tensorplay._composite_funcs. squeeze_copy ( input , dim = None ) [source] # tensorplay._composite_funcs.subtract URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.subtract.html tensorplay._composite_funcs.subtract tensorplay._composite_funcs. subtract ( input , other , * , alpha = 1 , out = None ) [source] # tensorplay._composite_funcs.t_copy URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.t_copy.html tensorplay._composite_funcs.t_copy tensorplay._composite_funcs. t_copy ( input ) [source] # tensorplay._composite_funcs.transpose_copy URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.transpose_copy.html tensorplay._composite_funcs.transpose_copy tensorplay._composite_funcs. transpose_copy ( input , dim0 , dim1 ) [source] # tensorplay._composite_funcs.trapezoid URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.trapezoid.html tensorplay._composite_funcs.trapezoid tensorplay._composite_funcs. trapezoid ( y , x = None , * , dx = None , dim = -1 ) [source] # tensorplay._composite_funcs.trapz URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.trapz.html tensorplay._composite_funcs.trapz tensorplay._composite_funcs. trapz ( y , x = None , * , dx = None , dim = -1 ) [source] # tensorplay._composite_funcs.tril_indices URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.tril_indices.html tensorplay._composite_funcs.tril_indices tensorplay._composite_funcs. tril_indices ( row , col , offset=0 , * , dtype= ) [source] # tensorplay._composite_funcs.triu_indices URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.triu_indices.html tensorplay._composite_funcs.triu_indices tensorplay._composite_funcs. triu_indices ( row , col , offset=0 , * , dtype= ) [source] # tensorplay._composite_funcs.true_divide URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.true_divide.html tensorplay._composite_funcs.true_divide tensorplay._composite_funcs. true_divide ( input , other , * , rounding_mode = None , out = None ) [source] # tensorplay._composite_funcs.unbind_copy URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.unbind_copy.html tensorplay._composite_funcs.unbind_copy tensorplay._composite_funcs. unbind_copy ( input , dim = 0 ) [source] # tensorplay._composite_funcs.unfold_copy URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.unfold_copy.html tensorplay._composite_funcs.unfold_copy tensorplay._composite_funcs. unfold_copy ( input , dimension , size , step ) [source] # tensorplay._composite_funcs.unique_consecutive URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.unique_consecutive.html tensorplay._composite_funcs.unique_consecutive tensorplay._composite_funcs. unique_consecutive ( input , return_inverse = False , return_counts = False , dim = None ) [source] # tensorplay._composite_funcs.unsqueeze_copy URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.unsqueeze_copy.html tensorplay._composite_funcs.unsqueeze_copy tensorplay._composite_funcs. unsqueeze_copy ( input , dim ) [source] # tensorplay._composite_funcs.vander URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.vander.html tensorplay._composite_funcs.vander tensorplay._composite_funcs. vander ( x , N = None , increasing = False ) [source] # tensorplay._composite_funcs.view_copy URL: https://www.tensorplay.cn/docs/generated/tensorplay._composite_funcs.view_copy.html tensorplay._composite_funcs.view_copy tensorplay._composite_funcs. view_copy ( input , size ) [source] # tensorplay._shape_funcs.broadcast_shapes URL: https://www.tensorplay.cn/docs/generated/tensorplay._shape_funcs.broadcast_shapes.html tensorplay._shape_funcs.broadcast_shapes tensorplay._shape_funcs. broadcast_shapes ( * shapes ) [source] Returns the broadcast shape of the given shapes (right-aligned). # tensorplay._shape_funcs.tensordot URL: https://www.tensorplay.cn/docs/generated/tensorplay._shape_funcs.tensordot.html tensorplay._shape_funcs.tensordot tensorplay._shape_funcs. tensordot ( input , other , dims = 2 ) [source] Contracts input and other over the given dimensions. dims may be a non-negative int (contract the last dims dims of input with the first dims dims of other ) or a pair (dims_a, dims_b) of dimension lists paired positionally. # tensorplay._shape_funcs.unravel_index URL: https://www.tensorplay.cn/docs/generated/tensorplay._shape_funcs.unravel_index.html tensorplay._shape_funcs.unravel_index tensorplay._shape_funcs. unravel_index ( indices , shape ) [source] Converts flat indices into coordinate tuples (one LongTensor per dim). Indices wrap around modulo the total size, matching torch semantics. # tensorplay.adaptive_avg_pool2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.adaptive_avg_pool2d.html tensorplay.adaptive_avg_pool2d tensorplay. adaptive_avg_pool2d ( input , output_size ) [source] # tensorplay.adaptive_avg_pool3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.adaptive_avg_pool3d.html tensorplay.adaptive_avg_pool3d tensorplay. adaptive_avg_pool3d ( input , output_size ) [source] # tensorplay.adaptive_max_pool2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.adaptive_max_pool2d.html tensorplay.adaptive_max_pool2d tensorplay. adaptive_max_pool2d ( input , output_size ) [source] # tensorplay.add_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.add_.html tensorplay.add_ tensorplay. add_ ( input , other , alpha = 1 ) [source] # tensorplay.add_relu URL: https://www.tensorplay.cn/docs/generated/tensorplay.add_relu.html tensorplay.add_relu tensorplay. add_relu ( input , other ) [source] # tensorplay.addcdiv_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.addcdiv_.html tensorplay.addcdiv_ tensorplay. addcdiv_ ( input , tensor1 , tensor2 , value = 1 ) [source] # tensorplay.addcmul_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.addcmul_.html tensorplay.addcmul_ tensorplay. addcmul_ ( input , tensor1 , tensor2 , value = 1 ) [source] # tensorplay.airy_ai URL: https://www.tensorplay.cn/docs/generated/tensorplay.airy_ai.html tensorplay.airy_ai tensorplay. airy_ai ( input ) [source] # autocast URL: https://www.tensorplay.cn/docs/generated/tensorplay.amp.autocast_mode.autocast.html autocast class tensorplay.amp.autocast_mode. autocast ( device_type : str , dtype : Any | None = None , enabled : bool = True , cache_enabled : bool | None = None ) [source] Instances of autocast serve as context managers or decorators that allow regions of your script to run in mixed precision. In these regions, ops run in an op-specific dtype chosen by autocast to improve performance while maintaining accuracy. When entering an autocast-enabled region, Tensors may be any type. You should not call half() or bfloat16() on your model(s) or inputs when using autocasting. autocast should wrap only the forward pass(es) of your network, including the loss computation(s). Backward passes under autocast are not recommended. Backward ops run in the same type that autocast used for corresponding forward ops. Example for CUDA Devices: # Creates model and optimizer in default precision model = Net () . cuda () optimizer = optim . SGD ( model . parameters (), ... ) for input , target in data : optimizer . zero_grad () # Enables autocasting for the forward pass (model + loss) with tensorplay . autocast ( device_type = "cuda" ): output = model ( input ) loss = loss_fn ( output , target ) # Exits the context manager before backward() loss . backward () optimizer . step () autocast can also be used as a decorator, e.g., on the forward method of your model: class AutocastModel ( nn . Module ): ... @tensorplay . autocast ( device_type = "cuda" ) def forward ( self , input ): ... Floating-point Tensors produced in an autocast-enabled region may be float16 . After returning to an autocast-disabled region, using them with floating-point Tensors of different dtypes may cause type mismatch errors. If so, cast the Tensor(s) produced in the autocast region back to float32 (or other dtype if desired). autocast(enabled=False) subregions can be nested in autocast-enabled regions. Locally disabling autocast can be useful, for example, if you want to force a subregion to run in a particular dtype . The autocast state is thread-local. If you want it enabled in a new thread, the context manager or decorator must be invoked in that thread. Parameters : device_type ( str , required ) – Device type to use. Possible values are: ‘cuda’ and ‘cpu’. The type is the same as the type attribute of a tensorplay.device . Thus, you may obtain the device type of a tensor using Tensor.device.type . enabled ( bool , optional ) – Whether autocasting should be enabled in the region. Default: True dtype ( tensorplay.dtype , optional ) – Data type for ops run in autocast. It uses the default value ( tensorplay.float16 for CUDA and tensorplay.bfloat16 for CPU), given by get_autocast_dtype() , if dtype is None . Default: None cache_enabled ( bool , optional ) – Whether the weight cache inside autocast should be enabled. Default: True # tensorplay.amp.autocast_mode.custom_bwd URL: https://www.tensorplay.cn/docs/generated/tensorplay.amp.autocast_mode.custom_bwd.html tensorplay.amp.autocast_mode.custom_bwd tensorplay.amp.autocast_mode. custom_bwd ( bwd = None , * , device_type : str ) [source] Create a helper decorator for backward methods of custom autograd functions. Autograd functions are subclasses of tensorplay.autograd.Function . Ensures that backward executes with the same autocast state as forward . Parameters : device_type ( str ) – Device type to use. ‘cuda’, ‘cpu’. The type is the same as the type attribute of a tensorplay.device . Thus, you may obtain the device type of a tensor using Tensor.device.type . # tensorplay.amp.autocast_mode.custom_fwd URL: https://www.tensorplay.cn/docs/generated/tensorplay.amp.autocast_mode.custom_fwd.html tensorplay.amp.autocast_mode.custom_fwd tensorplay.amp.autocast_mode. custom_fwd ( fwd = None , * , device_type : str , cast_inputs = None ) [source] Create a helper decorator for forward methods of custom autograd functions. Autograd functions are subclasses of tensorplay.autograd.Function . Parameters : device_type ( str ) – Device type to use. ‘cuda’, ‘cpu’. The type is the same as the type attribute of a tensorplay.device . Thus, you may obtain the device type of a tensor using Tensor.device.type . cast_inputs ( tensorplay.dtype or None, optional, default=None) – If not None , when forward runs in an autocast-enabled region, casts incoming floating-point Tensors to the target dtype (non-floating-point Tensors are not affected), then executes forward with autocast disabled. If None , forward ’s internal ops execute with the current autocast state. Note If the decorated forward is called outside an autocast-enabled region, custom_fwd is a no-op and cast_inputs has no effect. # tensorplay.amp.autocast_mode.get_autocast_cpu_dtype URL: https://www.tensorplay.cn/docs/generated/tensorplay.amp.autocast_mode.get_autocast_cpu_dtype.html tensorplay.amp.autocast_mode.get_autocast_cpu_dtype tensorplay.amp.autocast_mode. get_autocast_cpu_dtype ( ) [source] Return the dtype to be used for CPU autocasting. Warning Kept for backward compatibility. Prefer get_autocast_dtype() . # tensorplay.amp.autocast_mode.get_autocast_gpu_dtype URL: https://www.tensorplay.cn/docs/generated/tensorplay.amp.autocast_mode.get_autocast_gpu_dtype.html tensorplay.amp.autocast_mode.get_autocast_gpu_dtype tensorplay.amp.autocast_mode. get_autocast_gpu_dtype ( ) [source] Return the dtype to be used for CUDA autocasting. Warning Kept for backward compatibility. Prefer get_autocast_dtype() . # tensorplay.amp.autocast_mode.is_autocast_available URL: https://www.tensorplay.cn/docs/generated/tensorplay.amp.autocast_mode.is_autocast_available.html tensorplay.amp.autocast_mode.is_autocast_available tensorplay.amp.autocast_mode. is_autocast_available ( ) # GradScaler URL: https://www.tensorplay.cn/docs/generated/tensorplay.amp.grad_scaler.GradScaler.html GradScaler class tensorplay.amp.grad_scaler. GradScaler ( device : str = 'cuda' , init_scale : float = 65536.0 , growth_factor : float = 2.0 , backoff_factor : float = 0.5 , growth_interval : int = 2000 , enabled : bool = True ) [source] An instance scaler of GradScaler . Helps perform the steps of gradient scaling conveniently. scaler.scale(loss) multiplies a given loss by scaler ’s current scale factor. scaler.step(optimizer) safely unscales gradients and calls optimizer.step() . scaler.update() updates scaler ’s scale factor. Example: # Creates a GradScaler once at the beginning of training. scaler = GradScaler () for epoch in epochs : for input , target in data : optimizer . zero_grad () output = model ( input ) loss = loss_fn ( output , target ) # Scales loss. Calls backward() on scaled loss to create scaled gradients. scaler . scale ( loss ) . backward () # scaler.step() first unscales gradients of the optimizer's params. # If gradients don't contain infs/NaNs, optimizer.step() is then called, # otherwise, optimizer.step() is skipped. scaler . step ( optimizer ) # Updates the scale for next iteration. scaler . update () scaler dynamically estimates the scale factor each iteration. To minimize gradient underflow, a large scale factor should be used. However, float16 values can “overflow” (become inf or NaN) if the scale factor is too large. Therefore, the optimal scale factor is the largest factor that can be used without incurring inf or NaN gradient values. scaler approximates the optimal scale factor over time by checking the gradients for infs and NaNs during every scaler.step(optimizer) (or optional separate scaler.unscale_(optimizer) , see unscale_() ). If infs/NaNs are found, scaler.step(optimizer) skips the underlying optimizer.step() (so the params themselves remain uncorrupted) and update() multiplies the scale by backoff_factor . If no infs/NaNs are found, scaler.step(optimizer) runs the underlying optimizer.step() as usual. If growth_interval unskipped iterations occur consecutively, update() multiplies the scale by growth_factor . Parameters : device ( str , optional , default="cuda" ) – Device type to use. Possible values are: ‘cuda’ and ‘cpu’. The type is the same as the type attribute of a tensorplay.device . Thus, you may obtain the device type of a tensor using Tensor.device.type . init_scale ( float , optional , default=2.**16 ) – Initial scale factor. growth_factor ( float , optional , default=2.0 ) – Factor by which the scale is multiplied during update() if no inf/NaN gradients occur for growth_interval consecutive iterations. backoff_factor ( float , optional , default=0.5 ) – Factor by which the scale is multiplied during update() if inf/NaN gradients occur in an iteration. growth_interval ( int , optional , default=2000 ) – Number of consecutive iterations without inf/NaN gradients that must occur for the scale to be multiplied by growth_factor . enabled ( bool , optional ) – If False , disables gradient scaling. step() simply invokes the underlying optimizer.step() , and other methods become no-ops. Default: True get_backoff_factor ( ) → float [source] Return a Python float containing the scale backoff factor. get_growth_factor ( ) → float [source] Return a Python float containing the scale growth factor. get_growth_interval ( ) → int [source] Return a Python int containing the growth interval. get_scale ( ) → float [source] Return a Python float containing the current scale, or 1.0 if scaling is disabled. is_enabled ( ) → bool [source] Return a bool indicating whether this instance is enabled. load_state_dict ( state_dict : dict [ str , Any ] ) → None [source] Load the scaler state. If this instance is disabled, load_state_dict() is a no-op. Parameters : state_dict ( dict ) – scaler state. Should be an object returned from a call to state_dict() . scale ( outputs : TensorBase ) → TensorBase [source] scale ( outputs : list [ TensorBase ] ) → list [ TensorBase ] scale ( outputs : tuple [ TensorBase , ... ] ) → tuple [ TensorBase , ... ] scale ( outputs : Iterable [ tensorplay.Tensor ] ) → Iterable [ tensorplay.Tensor ] Multiplies (‘scales’) a tensor or list of tensors by the scale factor. Returns scaled outputs. If this instance of GradScaler is not enabled, outputs are returned unmodified. Parameters : outputs ( Tensor or iterable of Tensors ) – Outputs to scale. set_backoff_factor ( new_factor : float ) → None [source] Set a new scale backoff factor. Parameters : new_scale ( float ) – Value to use as the new scale backoff factor. set_growth_factor ( new_factor : float ) → None [source] Set a new scale growth factor. Parameters : new_scale ( float ) – Value to use as the new scale growth factor. set_growth_interval ( new_interval : int ) → None [source] Set a new growth interval. Parameters : new_interval ( int ) – Value to use as the new growth interval. state_dict ( ) → dict [ str , Any ] [source] Return the state of the scaler as a dict . It contains five entries: "scale" - a Python float containing the current scale "growth_factor" - a Python float containing the current growth factor "backoff_factor" - a Python float containing the current backoff factor "growth_interval" - a Python int containing the current growth interval "_growth_tracker" - a Python int containing the number of recent consecutive unskipped steps. If this instance is not enabled, returns an empty dict. Note If you wish to checkpoint the scaler’s state after a particular iteration, state_dict() should be called after update() . step ( optimizer : Optimizer , * args : Any , ** kwargs : Any ) → Any [source] Invoke unscale_(optimizer) followed by parameter update, if gradients are not infs/NaN. step() carries out the following two operations: Internally invokes unscale_(optimizer) (unless unscale_() was explicitly called for optimizer earlier in the iteration). As part of the unscale_() , gradients are checked for infs/NaNs. If no inf/NaN gradients are found, invokes optimizer.step() using the unscaled gradients. Otherwise, optimizer.step() is skipped to avoid corrupting the params. *args and **kwargs are forwarded to optimizer.step() . Returns the return value of optimizer.step(*args, **kwargs) . Parameters : optimizer ( tensorplay.optim.Optimizer ) – Optimizer that applies the gradients. args – Any arguments. kwargs – Any keyword arguments. Warning Closure use is not currently supported. unscale_ ( optimizer : Optimizer ) → None [source] Divides (“unscales”) the optimizer’s gradient tensors by the scale factor. unscale_() is optional, serving cases where you need to modify or inspect gradients between the backward pass(es) and step() . If unscale_() is not called explicitly, gradients will be unscaled automatically during step() . Simple example, using unscale_() to enable clipping of unscaled gradients: ... scaler . scale ( loss ) . backward () scaler . unscale_ ( optimizer ) tensorplay . nn . utils . clip_grad_norm_ ( model . parameters (), max_norm ) scaler . step ( optimizer ) scaler . update () Parameters : optimizer ( tensorplay.optim.Optimizer ) – Optimizer that owns the gradients to be unscaled. Warning unscale_() should only be called once per optimizer per step() call, and only after all gradients for that optimizer’s assigned parameters have been accumulated. Calling unscale_() twice for a given optimizer between each step() triggers a RuntimeError. update ( new_scale : float | TensorBase | None = None ) → None [source] Update the scale factor. If any optimizer steps were skipped the scale is multiplied by backoff_factor to reduce it. If growth_interval unskipped iterations occurred consecutively, the scale is multiplied by growth_factor to increase it. Passing new_scale sets the new scale value manually. ( new_scale is not used directly, it’s used to fill GradScaler’s internal scale tensor. So if new_scale was a tensor, later in-place changes to that tensor will not further affect the scale GradScaler uses internally.) Parameters : new_scale (float or tensorplay.Tensor , optional, default=None) – New scale factor. Warning update() should only be called at the end of the iteration, after scaler.step(optimizer) has been invoked for all optimizers used this iteration. # tensorplay.amp.is_autocast_available URL: https://www.tensorplay.cn/docs/generated/tensorplay.amp.is_autocast_available.html tensorplay.amp.is_autocast_available tensorplay.amp. is_autocast_available ( ) # tensorplay.are_deterministic_algorithms_enabled URL: https://www.tensorplay.cn/docs/generated/tensorplay.are_deterministic_algorithms_enabled.html tensorplay.are_deterministic_algorithms_enabled tensorplay. are_deterministic_algorithms_enabled ( ) → bool [source] Returns True if the global deterministic flag is turned on. Refer to tensorplay.use_deterministic_algorithms() documentation for more details. # tensorplay.as_tensor URL: https://www.tensorplay.cn/docs/generated/tensorplay.as_tensor.html tensorplay.as_tensor tensorplay. as_tensor ( data , dtype = None , device = None ) [source] Convert data into a tensor, sharing storage when possible. If data is already a tensor with the requested dtype and device, it is returned as-is (no copy). Otherwise it is converted, mirroring torch.as_tensor . Parameters : data ( tensor , list , or scalar ) – Initial data for the tensor. dtype ( tensorplay.DType , optional) – the desired data type of the returned tensor. device ( tensorplay.Device , str, optional) – the device of the constructed tensor. Example: >>> x = tensorplay . tensor ([ 1.0 , 2.0 ]) >>> tensorplay . as_tensor ( x ) is x True >>> tensorplay . as_tensor ([ 0 , 1 , 2 ], dtype = tensorplay . int64 ) . dtype == tensorplay . int64 True # tensorplay.autocast_decrement_nesting URL: https://www.tensorplay.cn/docs/generated/tensorplay.autocast_decrement_nesting.html tensorplay.autocast_decrement_nesting tensorplay. autocast_decrement_nesting ( ) → int # tensorplay.autocast_increment_nesting URL: https://www.tensorplay.cn/docs/generated/tensorplay.autocast_increment_nesting.html tensorplay.autocast_increment_nesting tensorplay. autocast_increment_nesting ( ) → int # autocast URL: https://www.tensorplay.cn/docs/generated/tensorplay.autocast.html autocast class tensorplay. autocast ( device_type : str , dtype : Any | None = None , enabled : bool = True , cache_enabled : bool | None = None ) [source] Instances of autocast serve as context managers or decorators that allow regions of your script to run in mixed precision. In these regions, ops run in an op-specific dtype chosen by autocast to improve performance while maintaining accuracy. When entering an autocast-enabled region, Tensors may be any type. You should not call half() or bfloat16() on your model(s) or inputs when using autocasting. autocast should wrap only the forward pass(es) of your network, including the loss computation(s). Backward passes under autocast are not recommended. Backward ops run in the same type that autocast used for corresponding forward ops. Example for CUDA Devices: # Creates model and optimizer in default precision model = Net () . cuda () optimizer = optim . SGD ( model . parameters (), ... ) for input , target in data : optimizer . zero_grad () # Enables autocasting for the forward pass (model + loss) with tensorplay . autocast ( device_type = "cuda" ): output = model ( input ) loss = loss_fn ( output , target ) # Exits the context manager before backward() loss . backward () optimizer . step () autocast can also be used as a decorator, e.g., on the forward method of your model: class AutocastModel ( nn . Module ): ... @tensorplay . autocast ( device_type = "cuda" ) def forward ( self , input ): ... Floating-point Tensors produced in an autocast-enabled region may be float16 . After returning to an autocast-disabled region, using them with floating-point Tensors of different dtypes may cause type mismatch errors. If so, cast the Tensor(s) produced in the autocast region back to float32 (or other dtype if desired). autocast(enabled=False) subregions can be nested in autocast-enabled regions. Locally disabling autocast can be useful, for example, if you want to force a subregion to run in a particular dtype . The autocast state is thread-local. If you want it enabled in a new thread, the context manager or decorator must be invoked in that thread. Parameters : device_type ( str , required ) – Device type to use. Possible values are: ‘cuda’ and ‘cpu’. The type is the same as the type attribute of a tensorplay.device . Thus, you may obtain the device type of a tensor using Tensor.device.type . enabled ( bool , optional ) – Whether autocasting should be enabled in the region. Default: True dtype ( tensorplay.dtype , optional ) – Data type for ops run in autocast. It uses the default value ( tensorplay.float16 for CUDA and tensorplay.bfloat16 for CPU), given by get_autocast_dtype() , if dtype is None . Default: None cache_enabled ( bool , optional ) – Whether the weight cache inside autocast should be enabled. Default: True # detect_anomaly URL: https://www.tensorplay.cn/docs/generated/tensorplay.autograd.anomaly_mode.detect_anomaly.html detect_anomaly class tensorplay.autograd.anomaly_mode. detect_anomaly ( check_nan = True ) [source] Context-manager that enables anomaly detection for the autograd engine. This does two things: Running the forward pass with detection enabled will allow the backward pass to print the traceback of the forward operation that created the failing backward function. If check_nan is True , any backward computation that generates “nan” value will raise an error. Default True . Warning This mode should be enabled only for debugging as the different tests will slow down your program execution. Example >>> import tensorplay >>> from tensorplay import autograd >>> class MyFunc ( autograd . Function ): ... @staticmethod ... def forward ( ctx , inp ): ... return inp . clone () ... ... @staticmethod ... def backward ( ctx , gO ): ... # Error during the backward pass ... raise RuntimeError ( "Some error in backward" ) ... return gO . clone () >>> def run_fn ( a ): ... out = MyFunc . apply ( a ) ... return out . sum () >>> inp = tensorplay . rand ( 10 , 10 , requires_grad = True ) >>> out = run_fn ( inp ) >>> out . backward () Traceback (most recent call last): File "", line 1, in out.backward() RuntimeError: Some error in backward >>> with autograd . detect_anomaly (): ... inp = tensorplay . rand ( 10 , 10 , requires_grad = True ) ... out = run_fn ( inp ) ... out . backward () Traceback of forward call that caused the error: File "tmp.py", line 53, in out = run_fn(inp) File "tmp.py", line 44, in run_fn out = MyFunc.apply(a) Traceback (most recent call last): File "", line 4, in RuntimeError: Some error in backward # set_detect_anomaly URL: https://www.tensorplay.cn/docs/generated/tensorplay.autograd.anomaly_mode.set_detect_anomaly.html set_detect_anomaly class tensorplay.autograd.anomaly_mode. set_detect_anomaly ( mode : bool , check_nan : bool = True ) [source] Context-manager that sets the anomaly detection for the autograd engine on or off. set_detect_anomaly will enable or disable the autograd anomaly detection based on its argument mode . It can be used as a context-manager or as a function. See detect_anomaly above for details of the anomaly detection behaviour. Parameters : mode ( bool ) – Flag whether to enable anomaly detection ( True ), or disable ( False ). check_nan ( bool ) – Flag whether to raise an error when the backward generate “nan” # tensorplay.autograd.backward URL: https://www.tensorplay.cn/docs/generated/tensorplay.autograd.backward.html tensorplay.autograd.backward tensorplay.autograd. backward ( tensors : collections.abc.Sequence [ tensorplay._C.TensorBase ] , grad_tensors : collections.abc.Sequence [ tensorplay._C.TensorBase ] | None = None , retain_graph : bool | None = None , create_graph : bool = False ) → None # Function URL: https://www.tensorplay.cn/docs/generated/tensorplay.autograd.function.Function.html Function class tensorplay.autograd.function. Function [source] Records operation history and defines formulas for differentiating ops. Supports two styles, mirroring torch.autograd.Function: Legacy style: forward(ctx, ...) / backward(ctx, ...) (forward receives a context object). Combined-forward style: define forward(*args, **kwargs) , setup_context(ctx, inputs, output) and use save_for_backward / save_for_forward inside setup_context instead of receiving a ctx argument in forward . classmethod apply ( * args , ** kwargs ) [source] Runs the operation and attaches gradient bookkeeping to outputs. static backward ( ctx , * grad_outputs ) [source] Defines a formula for differentiating the operation. static forward ( ctx , * args , ** kwargs ) [source] Performs the operation. This function is to be overridden by all subclasses. There are two ways to define forward: Usage 1 (Combined forward and ctx): @staticmethod def forward ( ctx , input1 , input2 ): ... return output Usage 2 (Separated forward and ctx): @staticmethod def forward ( input1 , input2 ): ... return output @staticmethod def setup_context ( ctx , inputs , output ): ... static jvp ( ctx , * grad_inputs ) [source] Defines a formula for computing the jacobian-vector product. Not yet supported by this engine; provided for API parity. static setup_context ( ctx , inputs , output ) [source] Sets up the context object (Usage 2 above). Parameters : ctx ( _Context ) – context object to modify in-place inputs ( tuple ) – inputs to forward() output ( Any ) – output of forward() static vmap ( info , in_dims , * args ) [source] Defines a formula for vectorizing the operation. Not yet supported by this engine; provided for API parity. # tensorplay.autograd.function.once_differentiable URL: https://www.tensorplay.cn/docs/generated/tensorplay.autograd.function.once_differentiable.html tensorplay.autograd.function.once_differentiable tensorplay.autograd.function. once_differentiable ( fn ) [source] Decorator to make a custom autograd Function’s backward run once, with gradients detached and grad-mode disabled inside. # tensorplay.autograd.functional.hessian URL: https://www.tensorplay.cn/docs/generated/tensorplay.autograd.functional.hessian.html tensorplay.autograd.functional.hessian tensorplay.autograd.functional. hessian ( func , inputs , create_graph = False , strict = False , vectorize = False , outer_jacobian_strategy = 'reverse-mode' ) [source] Compute the Hessian of a given scalar function. Parameters : func ( function ) – a Python function that takes Tensor inputs and returns a Tensor with a single element. inputs ( tuple of Tensors or Tensor ) – inputs to the function func . create_graph ( bool , optional ) – If True , the Hessian will be computed in a differentiable manner. Note that when strict is False , the result can not require gradients or be disconnected from the inputs. Defaults to False . strict ( bool , optional ) – If True , an error will be raised when we detect that there exists an input such that all the outputs are independent of it. If False , we return a Tensor of zeros as the hessian for said inputs, which is the expected mathematical value. Defaults to False . vectorize ( bool , optional ) – Not supported by this engine yet; passing True raises NotImplementedError . outer_jacobian_strategy ( str , optional ) – Only "reverse-mode" is supported; forward-mode AD raises NotImplementedError . Returns : if there is a single input, this will be a single Tensor containing the Hessian for the input. If it is a tuple, then the Hessian will be a tuple of tuples where Hessian[i][j] will contain the Hessian of the i th input and j th input with size the sum of the size of the i th input plus the size of the j th input. Hessian[i][j] will have the same dtype and device as the corresponding i th input. Return type : Hessian (Tensor or a tuple of tuple of Tensors) Example >>> def pow_reducer ( x ): ... return x . pow ( 3 ) . sum () >>> inputs = tensorplay . rand ( 2 , 2 ) >>> hessian ( pow_reducer , inputs ) tensor([[[[5.2265, 0.0000], [0.0000, 0.0000]], [[0.0000, 4.8221], [0.0000, 0.0000]]], [[[0.0000, 0.0000], [1.9456, 0.0000]], [[0.0000, 0.0000], [0.0000, 3.2550]]]]) # tensorplay.autograd.functional.hvp URL: https://www.tensorplay.cn/docs/generated/tensorplay.autograd.functional.hvp.html tensorplay.autograd.functional.hvp tensorplay.autograd.functional. hvp ( func , inputs , v = None , create_graph = False , strict = False ) [source] Compute the dot product between the scalar function’s Hessian and a vector v at a specified point. Parameters : func ( function ) – a Python function that takes Tensor inputs and returns a Tensor with a single element. inputs ( tuple of Tensors or Tensor ) – inputs to the function func . v ( tuple of Tensors or Tensor ) – The vector for which the Hessian vector product is computed. Must be the same size as the input of func . This argument is optional when func ’s input contains a single element and (if it is not provided) will be set as a Tensor containing a single 1 . create_graph ( bool , optional ) – If True , both the output and result will be computed in a differentiable way. Note that when strict is False , the result can not require gradients or be disconnected from the inputs. Defaults to False . strict ( bool , optional ) – If True , an error will be raised when we detect that there exists an input such that all the outputs are independent of it. If False , we return a Tensor of zeros as the hvp for said inputs, which is the expected mathematical value. Defaults to False . Returns : tuple with: func_output (tuple of Tensors or Tensor): output of func(inputs) hvp (tuple of Tensors or Tensor): result of the dot product with the same shape as the inputs. Return type : output ( tuple ) Example >>> def pow_reducer ( x ): ... return x . pow ( 3 ) . sum () >>> inputs = tensorplay . rand ( 2 , 2 ) >>> v = tensorplay . ones ( 2 , 2 ) >>> output = hvp ( pow_reducer , inputs , v ) >>> output [ 0 ] tensor(0.1448) >>> output [ 1 ] tensor([[2.0239, 1.6456], [2.4988, 1.4310]]) Note This function is significantly slower than vhp due to backward mode AD constraints. If your function is twice continuously differentiable, then hvp = vhp.t(). So if you know that your function satisfies this condition, you should use vhp instead that is much faster with the current implementation. # tensorplay.autograd.functional.jacobian URL: https://www.tensorplay.cn/docs/generated/tensorplay.autograd.functional.jacobian.html tensorplay.autograd.functional.jacobian tensorplay.autograd.functional. jacobian ( func , inputs , create_graph = False , strict = False , vectorize = False , strategy = 'reverse-mode' ) [source] Compute the Jacobian of a given function. Parameters : func ( function ) – a Python function that takes Tensor inputs and returns a tuple of Tensors or a Tensor. inputs ( tuple of Tensors or Tensor ) – inputs to the function func . create_graph ( bool , optional ) – If True , the Jacobian will be computed in a differentiable manner. Note that when strict is False , the result can not require gradients or be disconnected from the inputs. Defaults to False . strict ( bool , optional ) – If True , an error will be raised when we detect that there exists an input such that all the outputs are independent of it. If False , we return a Tensor of zeros as the jacobian for said inputs, which is the expected mathematical value. Defaults to False . vectorize ( bool , optional ) – Not supported by this engine yet; passing True raises NotImplementedError . strategy ( str , optional ) – Set to "reverse-mode" (default) or "forward-mode" . Forward-mode AD is not supported by this engine yet; passing it raises NotImplementedError . Returns : if there is a single input and output, this will be a single Tensor containing the Jacobian for the linearized inputs and output. If one of the two is a tuple, then the Jacobian will be a tuple of Tensors. If both of them are tuples, then the Jacobian will be a tuple of tuple of Tensors where Jacobian[i][j] will contain the Jacobian of the i th output and j th input and will have as size the concatenation of the sizes of the corresponding output and the corresponding input and will have same dtype and device as the corresponding input. Return type : Jacobian (Tensor or nested tuple of Tensors) Example >>> def exp_reducer ( x ): ... return x . exp () . sum ( dim = 1 ) >>> inputs = tensorplay . rand ( 2 , 2 ) >>> jacobian ( exp_reducer , inputs ) tensor([[[1.4917, 2.4352], [0.0000, 0.0000]], [[0.0000, 0.0000], [2.4369, 2.3799]]]) # tensorplay.autograd.functional.jvp URL: https://www.tensorplay.cn/docs/generated/tensorplay.autograd.functional.jvp.html tensorplay.autograd.functional.jvp tensorplay.autograd.functional. jvp ( func , inputs , v = None , create_graph = False , strict = False , mode = 'reversed' ) [source] Compute the dot product between the Jacobian of the given function at the point given by the inputs and a vector v . Parameters : func ( function ) – a Python function that takes Tensor inputs and returns a tuple of Tensors or a Tensor. inputs ( tuple of Tensors or Tensor ) – inputs to the function func . v ( tuple of Tensors or Tensor ) – The vector for which the Jacobian vector product is computed. Must be the same size as the input of func . This argument is optional when the input to func contains a single element and (if it is not provided) will be set as a Tensor containing a single 1 . create_graph ( bool , optional ) – If True , both the output and result will be computed in a differentiable way. Note that when strict is False , the result can not require gradients or be disconnected from the inputs. Defaults to False . strict ( bool , optional ) – If True , an error will be raised when we detect that there exists an input such that all the outputs are independent of it. If False , we return a Tensor of zeros as the jvp for said inputs, which is the expected mathematical value. Defaults to False . Returns : tuple with: func_output (tuple of Tensors or Tensor): output of func(inputs) jvp (tuple of Tensors or Tensor): result of the dot product with the same shape as the output. Return type : output ( tuple ) Note autograd.functional.jvp computes the jvp by using the backward of the backward (sometimes called the double backwards trick). This is not the most performant way of computing the jvp. Example >>> def exp_reducer ( x ): ... return x . exp () . sum ( dim = 1 ) >>> inputs = tensorplay . rand ( 4 , 4 ) >>> v = tensorplay . ones ( 4 , 4 ) >>> jvp ( exp_reducer , inputs , v ) (tensor([6.3090, 4.6742, 7.9114, 8.2106]), tensor([6.3090, 4.6742, 7.9114, 8.2106])) >>> def adder ( x , y ): ... return 2 * x + 3 * y >>> inputs = ( tensorplay . rand ( 2 ), tensorplay . rand ( 2 )) >>> v = ( tensorplay . ones ( 2 ), tensorplay . ones ( 2 )) >>> jvp ( adder , inputs , v ) (tensor([2.2399, 2.5005]), tensor([5., 5.])) mode (str, optional): “reversed” computes the jvp via the double backwards trick; “forward” uses native forward-mode AD kernels and propagates tangents in a single pass per op (requires func to be written with operators/methods supported by forward-mode, see tensorplay.autograd._forward ). Defaults to “reversed”. # tensorplay.autograd.functional.vhp URL: https://www.tensorplay.cn/docs/generated/tensorplay.autograd.functional.vhp.html tensorplay.autograd.functional.vhp tensorplay.autograd.functional. vhp ( func , inputs , v = None , create_graph = False , strict = False ) [source] Compute the dot product between vector v and Hessian of a given scalar function at a specified point. Parameters : func ( function ) – a Python function that takes Tensor inputs and returns a Tensor with a single element. inputs ( tuple of Tensors or Tensor ) – inputs to the function func . v ( tuple of Tensors or Tensor ) – The vector for which the vector Hessian product is computed. Must be the same size as the input of func . This argument is optional when func ’s input contains a single element and (if it is not provided) will be set as a Tensor containing a single 1 . create_graph ( bool , optional ) – If True , both the output and result will be computed in a differentiable way. Note that when strict is False , the result can not require gradients or be disconnected from the inputs. Defaults to False . strict ( bool , optional ) – If True , an error will be raised when we detect that there exists an input such that all the outputs are independent of it. If False , we return a Tensor of zeros as the vhp for said inputs, which is the expected mathematical value. Defaults to False . Returns : tuple with: func_output (tuple of Tensors or Tensor): output of func(inputs) vhp (tuple of Tensors or Tensor): result of the dot product with the same shape as the inputs. Return type : output ( tuple ) Example >>> def pow_reducer ( x ): ... return x . pow ( 3 ) . sum () >>> inputs = tensorplay . rand ( 2 , 2 ) >>> v = tensorplay . ones ( 2 , 2 ) >>> output = vhp ( pow_reducer , inputs , v ) >>> output [ 0 ] tensor(0.5591) >>> output [ 1 ] tensor([[1.0689, 1.2431], [3.0989, 4.4456]]) # tensorplay.autograd.functional.vjp URL: https://www.tensorplay.cn/docs/generated/tensorplay.autograd.functional.vjp.html tensorplay.autograd.functional.vjp tensorplay.autograd.functional. vjp ( func , inputs , v = None , create_graph = False , strict = False ) [source] Compute the dot product between a vector v and the Jacobian of the given function at the point given by the inputs. Parameters : func ( function ) – a Python function that takes Tensor inputs and returns a tuple of Tensors or a Tensor. inputs ( tuple of Tensors or Tensor ) – inputs to the function func . v ( tuple of Tensors or Tensor ) – The vector for which the vector Jacobian product is computed. Must be the same size as the output of func . This argument is optional when the output of func contains a single element and (if it is not provided) will be set as a Tensor containing a single 1 . create_graph ( bool , optional ) – If True , both the output and result will be computed in a differentiable way. Note that when strict is False , the result can not require gradients or be disconnected from the inputs. Defaults to False . strict ( bool , optional ) – If True , an error will be raised when we detect that there exists an input such that all the outputs are independent of it. If False , we return a Tensor of zeros as the vjp for said inputs, which is the expected mathematical value. Defaults to False . Returns : tuple with: func_output (tuple of Tensors or Tensor): output of func(inputs) vjpval (tuple of Tensors or Tensor): result of the dot product with the same shape as the inputs. Return type : output ( tuple ) Example >>> def exp_reducer ( x ): ... return x . exp () . sum ( dim = 1 ) >>> inputs = tensorplay . rand ( 4 , 4 ) >>> v = tensorplay . ones ( 4 ) >>> vjp ( exp_reducer , inputs , v ) (tensor([5.7817, 7.2458, 5.7830, 6.7782]), tensor([[1.4458, 1.3962, 1.3042, 1.6354], [2.1288, 1.0652, 1.5483, 2.5035], [2.2046, 1.1292, 1.1432, 1.3059], [1.3225, 1.6652, 1.7753, 2.0152]])) # enable_grad URL: https://www.tensorplay.cn/docs/generated/tensorplay.autograd.grad_mode.enable_grad.html enable_grad class tensorplay.autograd.grad_mode. enable_grad ( orig_func = None ) [source] Context-manager that enables gradient calculation. Enables gradient calculation, if it has been disabled via no_grad or set_grad_enabled . This context manager is thread local; it will not affect computation in other threads. Also functions as a decorator. Note enable_grad is one of several mechanisms that can enable or disable gradients locally see Locally disabling gradient computation for more information on how they compare. Note This API does not apply to forward-mode AD . Example:: >>> # xdoctest: +SKIP >>> x = tensorplay . tensor ([ 1. ], requires_grad = True ) >>> with tensorplay . no_grad (): ... with tensorplay . enable_grad (): ... y = x * 2 >>> y . requires_grad True >>> y . backward () >>> x . grad tensor([2.]) >>> @tensorplay . enable_grad () ... def doubler ( x ): ... return x * 2 >>> with tensorplay . no_grad (): ... z = doubler ( x ) >>> z . requires_grad True >>> @tensorplay . enable_grad () ... def tripler ( x ): ... return x * 3 >>> with tensorplay . no_grad (): ... z = tripler ( x ) >>> z . requires_grad True # inference_mode URL: https://www.tensorplay.cn/docs/generated/tensorplay.autograd.grad_mode.inference_mode.html inference_mode class tensorplay.autograd.grad_mode. inference_mode ( mode = True ) [source] Context manager that enables or disables inference mode. InferenceMode is analogous to no_grad and should be used when you are certain your operations will not interact with autograd (e.g., during data loading or model evaluation). Compared to no_grad , it removes additional overhead by disabling view tracking and version counter bumps. It is also more restrictive, in that tensors created in this mode cannot be used in computations recorded by autograd. This context manager is thread-local; it does not affect computation in other threads. Also functions as a decorator. Note Inference mode is one of several mechanisms that can locally enable or disable gradients. See Locally disabling gradient computation for a comparison. If avoiding the use of tensors created in inference mode in autograd-tracked regions is difficult, consider benchmarking your code with and without inference mode to weigh the performance benefits against the trade-offs. You can always use no_grad instead. Note Unlike some other mechanisms that locally enable or disable grad, entering inference_mode also disables forward-mode AD . Parameters : mode ( bool or function ) – Either a boolean flag to enable or disable inference mode, or a Python function to decorate with inference mode enabled. Example:: >>> import tensorplay >>> x = tensorplay . ones ( 1 , 2 , 3 , requires_grad = True ) >>> with tensorplay . inference_mode (): ... y = x * x >>> y . requires_grad False >>> y . _version Traceback (most recent call last): File "", line 1, in RuntimeError : Inference tensors do not track version counter. >>> @tensorplay . inference_mode () ... def func ( x ): ... return x * x >>> out = func ( x ) >>> out . requires_grad False >>> @tensorplay . inference_mode () ... def doubler ( x ): ... return x * 2 >>> out = doubler ( x ) >>> out . requires_grad False clone ( ) → inference_mode [source] Create a copy of this class # tensorplay.autograd.grad_mode.is_grad_enabled URL: https://www.tensorplay.cn/docs/generated/tensorplay.autograd.grad_mode.is_grad_enabled.html tensorplay.autograd.grad_mode.is_grad_enabled tensorplay.autograd.grad_mode. is_grad_enabled ( ) [source] # no_grad URL: https://www.tensorplay.cn/docs/generated/tensorplay.autograd.grad_mode.no_grad.html no_grad class tensorplay.autograd.grad_mode. no_grad ( orig_func = None ) [source] Context-manager that disables gradient calculation. Disabling gradient calculation is useful for inference, when you are sure that you will not call Tensor.backward() . It will reduce memory consumption for computations that would otherwise have requires_grad=True . In this mode, the result of every computation will have requires_grad=False , even when the inputs have requires_grad=True . There is an exception! All factory functions, or functions that create a new Tensor and take a requires_grad kwarg, will NOT be affected by this mode. This context manager is thread local; it will not affect computation in other threads. Also functions as a decorator. Note No-grad is one of several mechanisms that can enable or disable gradients locally see Locally disabling gradient computation for more information on how they compare. Note This API does not apply to forward-mode AD . If you want to disable forward AD for a computation, you can unpack your dual tensors. Example:: >>> x = tensorplay . tensor ([ 1. ], requires_grad = True ) >>> with tensorplay . no_grad (): ... y = x * 2 >>> y . requires_grad False >>> @tensorplay . no_grad () ... def doubler ( x ): ... return x * 2 >>> z = doubler ( x ) >>> z . requires_grad False >>> @tensorplay . no_grad () ... def tripler ( x ): ... return x * 3 >>> z = tripler ( x ) >>> z . requires_grad False >>> # factory function exception >>> with tensorplay . no_grad (): ... a = tensorplay . nn . Parameter ( tensorplay . rand ( 10 )) >>> a . requires_grad True # set_grad_enabled URL: https://www.tensorplay.cn/docs/generated/tensorplay.autograd.grad_mode.set_grad_enabled.html set_grad_enabled class tensorplay.autograd.grad_mode. set_grad_enabled ( mode : bool ) [source] Context-manager that sets gradient calculation on or off. set_grad_enabled will enable or disable grads based on its argument mode . It can be used as a context-manager or as a function. This context manager is thread local; it will not affect computation in other threads. Parameters : mode ( bool ) – Flag whether to enable grad ( True ), or disable ( False ). This can be used to conditionally enable gradients. Note set_grad_enabled is one of several mechanisms that can enable or disable gradients locally see Locally disabling gradient computation for more information on how they compare. Note This API does not apply to forward-mode AD . Example:: >>> # xdoctest: +SKIP >>> x = tensorplay . tensor ([ 1. ], requires_grad = True ) >>> is_train = False >>> with tensorplay . set_grad_enabled ( is_train ): ... y = x * 2 >>> y . requires_grad False >>> _ = tensorplay . set_grad_enabled ( True ) >>> y = x * 2 >>> y . requires_grad True >>> _ = tensorplay . set_grad_enabled ( False ) >>> y = x * 2 >>> y . requires_grad False clone ( ) → set_grad_enabled [source] Create a copy of this class # tensorplay.autograd.grad URL: https://www.tensorplay.cn/docs/generated/tensorplay.autograd.grad.html tensorplay.autograd.grad tensorplay.autograd. grad ( outputs : TensorBase | Sequence [ TensorBase ] , inputs : TensorBase | Sequence [ TensorBase ] , grad_outputs : TensorBase | Sequence [ TensorBase ] | None = None , retain_graph : bool | None = None , create_graph : bool = False , allow_unused : bool | None = None ) → tuple [ TensorBase | None , ... ] [source] Compute and return the sum of gradients of outputs with respect to the inputs. grad_outputs should be a sequence of length matching output containing the “vector” in vector-Jacobian product, usually the pre-computed gradients w.r.t. each of the outputs. If an output doesn’t require_grad, then the gradient can be None ). Note If you run any forward ops, create grad_outputs , and/or call grad in a user-specified CUDA stream context, see Stream semantics of backward passes . Parameters : outputs ( sequence of Tensor or GradientEdge ) – outputs of the differentiated function. inputs ( sequence of Tensor or GradientEdge ) – Inputs w.r.t. which the gradient will be returned (and not accumulated into .grad ). grad_outputs ( sequence of Tensor ) – The “vector” in the vector-Jacobian product. Usually gradients w.r.t. each output. None values can be specified for scalar Tensors or ones that don’t require grad. If a None value would be acceptable for all grad_tensors, then this argument is optional. Default: None. retain_graph ( bool , optional ) – If False , the graph used to compute the grad will be freed. Note that in nearly all cases setting this option to True is not needed and often can be worked around in a much more efficient way. Defaults to the value of create_graph . create_graph ( bool , optional ) – If True , graph of the derivative will be constructed, allowing to compute higher order derivative products. Default: False . allow_unused ( Optional [ bool ] , optional ) – If False , specifying inputs that were not used when computing outputs (and therefore their grad is always zero) is an error. Defaults to the value of materialize_grads . # tensorplay.autograd.gradcheck.get_analytical_jacobian URL: https://www.tensorplay.cn/docs/generated/tensorplay.autograd.gradcheck.get_analytical_jacobian.html tensorplay.autograd.gradcheck.get_analytical_jacobian tensorplay.autograd.gradcheck. get_analytical_jacobian ( inputs , output , nondet_tol = 0.0 , grad_out = 1.0 ) [source] # tensorplay.autograd.gradcheck.get_numerical_jacobian_wrt_specific_input URL: https://www.tensorplay.cn/docs/generated/tensorplay.autograd.gradcheck.get_numerical_jacobian_wrt_specific_input.html tensorplay.autograd.gradcheck.get_numerical_jacobian_wrt_specific_input tensorplay.autograd.gradcheck. get_numerical_jacobian_wrt_specific_input ( fn , input_idx , inputs , outputs , eps , input = None ) → tuple [source] # tensorplay.autograd.gradcheck.get_numerical_jacobian URL: https://www.tensorplay.cn/docs/generated/tensorplay.autograd.gradcheck.get_numerical_jacobian.html tensorplay.autograd.gradcheck.get_numerical_jacobian tensorplay.autograd.gradcheck. get_numerical_jacobian ( fn , inputs , target = None , eps = 0.001 , grad_out = 1.0 ) [source] Compute the numerical Jacobian for a given fn and its inputs. Warning get_numerical_jacobian was part of PyTorch’s private API and is kept only for backward compatibility. Parameters : fn – the function to compute the Jacobian for (must take inputs as a tuple) inputs – input to fn target – the Tensors wrt whom Jacobians are calculated (default=`input`) eps – the magnitude of the perturbation during finite differencing (default=`1e-3`) grad_out – defaults to 1.0. Returns : A list of Jacobians of fn (restricted to its first output) with respect to each input or target, if provided. Note that target may not even be part of input to fn , so please be very careful in this to not clone target . # tensorplay.autograd.gradcheck.gradcheck URL: https://www.tensorplay.cn/docs/generated/tensorplay.autograd.gradcheck.gradcheck.html tensorplay.autograd.gradcheck.gradcheck tensorplay.autograd.gradcheck. gradcheck ( func , inputs , * , eps : float = 1e-06 , atol : float = 1e-05 , rtol : float = 0.001 , raise_exception : bool = True , nondet_tol : float = 0.0 , check_undefined_grad : bool = True , check_grad_dtypes : bool = False , check_batched_grad : bool = False , check_batched_forward_grad : bool = False , check_forward_ad : bool = False , check_backward_ad : bool = True , fast_mode : bool = False , masked : bool | None = None ) → bool [source] Check gradients computed via small finite differences against analytical gradients wrt tensors in inputs that are of floating point or complex type and with requires_grad=True . The check between numerical and analytical gradients uses allclose() . Note The default values are designed for input of double precision. This check will likely fail if input is of less precision, e.g., FloatTensor . Note Gradcheck may fail when evaluated on non-differentiable points because the numerically computed gradients via finite differencing may differ those computed analytically (not necessarily because either is incorrect). Warning If any checked tensor in input has overlapping memory, i.e., different indices pointing to the same memory address (e.g., from expand ), this check will likely fail because the numerical gradients computed by point perturbation at such indices will change values at all other indices that share the same memory address. Parameters : func ( function ) – a Python function that takes Tensor inputs and returns a Tensor or a tuple of Tensors inputs ( tuple of Tensor or Tensor ) – inputs to the function eps ( float , optional ) – perturbation for finite differences atol ( float , optional ) – absolute tolerance rtol ( float , optional ) – relative tolerance raise_exception ( bool , optional ) – indicating whether to raise an exception if the check fails. The exception gives more information about the exact nature of the failure. This is helpful when debugging gradchecks. nondet_tol ( float , optional ) – tolerance for non-determinism. When running identical inputs through the differentiation, the results must either match exactly (default, 0.0) or be within this tolerance. check_undefined_grad ( bool , optional ) – if True , check if undefined output grads are supported and treated as zeros, for Tensor outputs. check_grad_dtypes ( bool , optional ) – if True , check that the gradient dtypes match the ones from the numerical computation. Defaults to False . check_batched_grad ( bool , optional ) – Not supported by this engine yet; True raises NotImplementedError . Defaults to False. check_batched_forward_grad ( bool , optional ) – Requires forward AD; True raises NotImplementedError . Defaults to False. check_forward_ad ( bool , optional ) – Requires forward AD; True raises NotImplementedError . Defaults to False. check_backward_ad ( bool , optional ) – if False , do not perform any checks that rely on backward mode AD to be implemented. Defaults to True . fast_mode ( bool , optional ) – Only the slow implementation exists in this engine; True raises NotImplementedError . Defaults to False. masked ( bool , optional ) – Kept for signature parity with torch; has no effect since this engine has no sparse layouts. Defaults to False . Returns : True if all differences satisfy allclose condition # tensorplay.autograd.gradcheck.GradcheckError URL: https://www.tensorplay.cn/docs/generated/tensorplay.autograd.gradcheck.GradcheckError.html tensorplay.autograd.gradcheck.GradcheckError exception tensorplay.autograd.gradcheck. GradcheckError [source] Error raised by gradcheck() and gradgradcheck() . # tensorplay.autograd.gradcheck.gradgradcheck URL: https://www.tensorplay.cn/docs/generated/tensorplay.autograd.gradcheck.gradgradcheck.html tensorplay.autograd.gradcheck.gradgradcheck tensorplay.autograd.gradcheck. gradgradcheck ( func , inputs , grad_outputs = None , * , eps : float = 1e-06 , atol : float = 1e-05 , rtol : float = 0.001 , gen_non_contig_grad_outputs : bool = False , raise_exception : bool = True , nondet_tol : float = 0.0 , check_undefined_grad : bool = True , check_grad_dtypes : bool = False , check_batched_grad : bool = False , check_fwd_over_rev : bool = False , check_rev_over_rev : bool = True , fast_mode : bool = False , masked : bool = False ) → bool [source] Check gradients of gradients computed via small finite differences against analytical gradients wrt tensors in inputs and grad_outputs that are of floating point or complex type and with requires_grad=True . This function checks that backpropagating through the gradients computed to the given grad_outputs are correct. The check between numerical and analytical gradients uses allclose() . Note The default values are designed for input and grad_outputs of double precision. This check will likely fail if they are of less precision, e.g., FloatTensor . Parameters : func ( function ) – a Python function that takes Tensor inputs and returns a Tensor or a tuple of Tensors inputs ( tuple of Tensor or Tensor ) – inputs to the function grad_outputs ( tuple of [ Tensor or None ] or Tensor , optional ) – The gradients with respect to the function’s outputs. eps ( float , optional ) – perturbation for finite differences atol ( float , optional ) – absolute tolerance rtol ( float , optional ) – relative tolerance gen_non_contig_grad_outputs ( bool , optional ) – Not supported by this engine yet; True raises NotImplementedError . raise_exception ( bool , optional ) – indicating whether to raise an exception if the check fails. The exception gives more information about the exact nature of the failure. This is helpful when debugging gradchecks. nondet_tol ( float , optional ) – tolerance for non-determinism. When running identical inputs through the differentiation, the results must either match exactly (default, 0.0) or be within this tolerance. Note that a small amount of nondeterminism in the gradient will lead to larger inaccuracies in the second derivative. check_undefined_grad ( bool , optional ) – if True, check if undefined output grads are supported and treated as zeros check_batched_grad ( bool , optional ) – Not supported by this engine yet. fast_mode ( bool , optional ) – Not supported by this engine yet. masked ( bool , optional ) – Kept for signature parity with torch. Returns : True if all differences satisfy allclose condition # tensorplay.autograd.jvp URL: https://www.tensorplay.cn/docs/generated/tensorplay.autograd.jvp.html tensorplay.autograd.jvp tensorplay.autograd. jvp ( func , inputs , v = None , create_graph = False , strict = False , mode = 'reversed' ) [source] Compute the dot product between the Jacobian of the given function at the point given by the inputs and a vector v . Parameters : func ( function ) – a Python function that takes Tensor inputs and returns a tuple of Tensors or a Tensor. inputs ( tuple of Tensors or Tensor ) – inputs to the function func . v ( tuple of Tensors or Tensor ) – The vector for which the Jacobian vector product is computed. Must be the same size as the input of func . This argument is optional when the input to func contains a single element and (if it is not provided) will be set as a Tensor containing a single 1 . create_graph ( bool , optional ) – If True , both the output and result will be computed in a differentiable way. Note that when strict is False , the result can not require gradients or be disconnected from the inputs. Defaults to False . strict ( bool , optional ) – If True , an error will be raised when we detect that there exists an input such that all the outputs are independent of it. If False , we return a Tensor of zeros as the jvp for said inputs, which is the expected mathematical value. Defaults to False . Returns : tuple with: func_output (tuple of Tensors or Tensor): output of func(inputs) jvp (tuple of Tensors or Tensor): result of the dot product with the same shape as the output. Return type : output ( tuple ) Note autograd.functional.jvp computes the jvp by using the backward of the backward (sometimes called the double backwards trick). This is not the most performant way of computing the jvp. Example >>> def exp_reducer ( x ): ... return x . exp () . sum ( dim = 1 ) >>> inputs = tensorplay . rand ( 4 , 4 ) >>> v = tensorplay . ones ( 4 , 4 ) >>> jvp ( exp_reducer , inputs , v ) (tensor([6.3090, 4.6742, 7.9114, 8.2106]), tensor([6.3090, 4.6742, 7.9114, 8.2106])) >>> def adder ( x , y ): ... return 2 * x + 3 * y >>> inputs = ( tensorplay . rand ( 2 ), tensorplay . rand ( 2 )) >>> v = ( tensorplay . ones ( 2 ), tensorplay . ones ( 2 )) >>> jvp ( adder , inputs , v ) (tensor([2.2399, 2.5005]), tensor([5., 5.])) mode (str, optional): “reversed” computes the jvp via the double backwards trick; “forward” uses native forward-mode AD kernels and propagates tangents in a single pass per op (requires func to be written with operators/methods supported by forward-mode, see tensorplay.autograd._forward ). Defaults to “reversed”. # tensorplay.avg_pool2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.avg_pool2d.html tensorplay.avg_pool2d tensorplay. avg_pool2d ( input , kernel_size , stride = [] , padding = [0, 0] , ceil_mode = False , count_include_pad = True , divisor_override = None ) [source] # tensorplay.avg_pool3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.avg_pool3d.html tensorplay.avg_pool3d tensorplay. avg_pool3d ( input , kernel_size , stride = [] , padding = [0, 0] , ceil_mode = False , count_include_pad = True , divisor_override = None ) [source] # tensorplay.batch_norm URL: https://www.tensorplay.cn/docs/generated/tensorplay.batch_norm.html tensorplay.batch_norm tensorplay. batch_norm ( input , weight , bias , running_mean , running_var , training , momentum , eps ) [source] # tensorplay.bernoulli_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.bernoulli_.html tensorplay.bernoulli_ tensorplay. bernoulli_ ( input ) [source] # tensorplay.bessel_j0 URL: https://www.tensorplay.cn/docs/generated/tensorplay.bessel_j0.html tensorplay.bessel_j0 tensorplay. bessel_j0 ( input ) [source] # tensorplay.bessel_j1 URL: https://www.tensorplay.cn/docs/generated/tensorplay.bessel_j1.html tensorplay.bessel_j1 tensorplay. bessel_j1 ( input ) [source] # tensorplay.bessel_y0 URL: https://www.tensorplay.cn/docs/generated/tensorplay.bessel_y0.html tensorplay.bessel_y0 tensorplay. bessel_y0 ( input ) [source] # tensorplay.bessel_y1 URL: https://www.tensorplay.cn/docs/generated/tensorplay.bessel_y1.html tensorplay.bessel_y1 tensorplay. bessel_y1 ( input ) [source] # tensorplay.binary_cross_entropy_with_logits URL: https://www.tensorplay.cn/docs/generated/tensorplay.binary_cross_entropy_with_logits.html tensorplay.binary_cross_entropy_with_logits tensorplay. binary_cross_entropy_with_logits ( input , target , weight = None , pos_weight = None ) [source] # tensorplay.binary_cross_entropy URL: https://www.tensorplay.cn/docs/generated/tensorplay.binary_cross_entropy.html tensorplay.binary_cross_entropy tensorplay. binary_cross_entropy ( input , target ) [source] # tensorplay.cauchy_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.cauchy_.html tensorplay.cauchy_ tensorplay. cauchy_ ( input , median = 0.0 , sigma = 1.0 ) [source] # tensorplay.celu URL: https://www.tensorplay.cn/docs/generated/tensorplay.celu.html tensorplay.celu tensorplay. celu ( input , alpha = 1.0 ) [source] # tensorplay.chebyshev_polynomial_t URL: https://www.tensorplay.cn/docs/generated/tensorplay.chebyshev_polynomial_t.html tensorplay.chebyshev_polynomial_t tensorplay. chebyshev_polynomial_t ( x , n ) [source] # tensorplay.chebyshev_polynomial_u URL: https://www.tensorplay.cn/docs/generated/tensorplay.chebyshev_polynomial_u.html tensorplay.chebyshev_polynomial_u tensorplay. chebyshev_polynomial_u ( x , n ) [source] # tensorplay.chebyshev_polynomial_v URL: https://www.tensorplay.cn/docs/generated/tensorplay.chebyshev_polynomial_v.html tensorplay.chebyshev_polynomial_v tensorplay. chebyshev_polynomial_v ( x , n ) [source] # tensorplay.chebyshev_polynomial_w URL: https://www.tensorplay.cn/docs/generated/tensorplay.chebyshev_polynomial_w.html tensorplay.chebyshev_polynomial_w tensorplay. chebyshev_polynomial_w ( x , n ) [source] # tensorplay.cholesky URL: https://www.tensorplay.cn/docs/generated/tensorplay.cholesky.html tensorplay.cholesky tensorplay. cholesky ( input , upper = False ) [source] # tensorplay.circular_pad_nd URL: https://www.tensorplay.cn/docs/generated/tensorplay.circular_pad_nd.html tensorplay.circular_pad_nd tensorplay. circular_pad_nd ( input , pad ) [source] # tensorplay.clamp_max URL: https://www.tensorplay.cn/docs/generated/tensorplay.clamp_max.html tensorplay.clamp_max tensorplay. clamp_max ( input , max ) [source] # tensorplay.clamp_min URL: https://www.tensorplay.cn/docs/generated/tensorplay.clamp_min.html tensorplay.clamp_min tensorplay. clamp_min ( input , min ) [source] # tensorplay.clear_autocast_cache URL: https://www.tensorplay.cn/docs/generated/tensorplay.clear_autocast_cache.html tensorplay.clear_autocast_cache tensorplay. clear_autocast_cache ( ) → None # tensorplay.col2im URL: https://www.tensorplay.cn/docs/generated/tensorplay.col2im.html tensorplay.col2im tensorplay. col2im ( input , output_size , kernel_size , dilation = [] , padding = [] , stride = [] ) [source] # tensorplay.compiler.api.compile URL: https://www.tensorplay.cn/docs/generated/tensorplay.compiler.api.compile.html tensorplay.compiler.api.compile tensorplay.compiler.api. compile ( model : Callable [ [ ... ] , Any ] | None = None , * , fullgraph : bool = False , dynamic : bool | None = None , backend : str | Callable [ [ ... ] , CompiledFn ] | None = None , mode : str | None = None , options : dict [ str , Any ] | None = None , name : str | None = None , disable : bool = False , recompile_limit : int | None = None , isolate_recompiles : bool = False , strict_native : bool = False , dynamic_shapes : Any | None = None ) → Callable [ [ ... ] , Any ] [source] Compile a callable through the TensorPlay compiler frontend. backend may be a registered name or a callable with the contract backend(graph_module, example_inputs, **kwargs) -> callable . The frontend caches specializations by input metadata and falls back to the original callable when a non- fullgraph capture cannot represent the Python region. # tensorplay.constant_pad_nd URL: https://www.tensorplay.cn/docs/generated/tensorplay.constant_pad_nd.html tensorplay.constant_pad_nd tensorplay. constant_pad_nd ( input , pad , value ) [source] # tensorplay.contiguous URL: https://www.tensorplay.cn/docs/generated/tensorplay.contiguous.html tensorplay.contiguous tensorplay. contiguous ( input , memory_format = MemoryFormat.CONTIGUOUS ) [source] # tensorplay.conv_transpose1d_grad_bias URL: https://www.tensorplay.cn/docs/generated/tensorplay.conv_transpose1d_grad_bias.html tensorplay.conv_transpose1d_grad_bias tensorplay. conv_transpose1d_grad_bias ( grad_output , input , weight , stride , padding , output_padding , groups , dilation ) [source] # tensorplay.conv_transpose1d_grad_input URL: https://www.tensorplay.cn/docs/generated/tensorplay.conv_transpose1d_grad_input.html tensorplay.conv_transpose1d_grad_input tensorplay. conv_transpose1d_grad_input ( grad_output , input , weight , stride , padding , output_padding , groups , dilation ) [source] # tensorplay.conv_transpose1d_grad_weight URL: https://www.tensorplay.cn/docs/generated/tensorplay.conv_transpose1d_grad_weight.html tensorplay.conv_transpose1d_grad_weight tensorplay. conv_transpose1d_grad_weight ( grad_output , input , weight , stride , padding , output_padding , groups , dilation ) [source] # tensorplay.conv_transpose2d_grad_bias URL: https://www.tensorplay.cn/docs/generated/tensorplay.conv_transpose2d_grad_bias.html tensorplay.conv_transpose2d_grad_bias tensorplay. conv_transpose2d_grad_bias ( grad_output , input , weight , stride , padding , output_padding , groups , dilation ) [source] # tensorplay.conv_transpose2d_grad_input URL: https://www.tensorplay.cn/docs/generated/tensorplay.conv_transpose2d_grad_input.html tensorplay.conv_transpose2d_grad_input tensorplay. conv_transpose2d_grad_input ( grad_output , input , weight , stride , padding , output_padding , groups , dilation ) [source] # tensorplay.conv_transpose2d_grad_weight URL: https://www.tensorplay.cn/docs/generated/tensorplay.conv_transpose2d_grad_weight.html tensorplay.conv_transpose2d_grad_weight tensorplay. conv_transpose2d_grad_weight ( grad_output , input , weight , stride , padding , output_padding , groups , dilation ) [source] # tensorplay.conv_transpose3d_grad_bias URL: https://www.tensorplay.cn/docs/generated/tensorplay.conv_transpose3d_grad_bias.html tensorplay.conv_transpose3d_grad_bias tensorplay. conv_transpose3d_grad_bias ( grad_output , input , weight , stride , padding , output_padding , groups , dilation ) [source] # tensorplay.conv_transpose3d_grad_input URL: https://www.tensorplay.cn/docs/generated/tensorplay.conv_transpose3d_grad_input.html tensorplay.conv_transpose3d_grad_input tensorplay. conv_transpose3d_grad_input ( grad_output , input , weight , stride , padding , output_padding , groups , dilation ) [source] # tensorplay.conv_transpose3d_grad_weight URL: https://www.tensorplay.cn/docs/generated/tensorplay.conv_transpose3d_grad_weight.html tensorplay.conv_transpose3d_grad_weight tensorplay. conv_transpose3d_grad_weight ( grad_output , input , weight , stride , padding , output_padding , groups , dilation ) [source] # tensorplay.conv1d_grad_bias URL: https://www.tensorplay.cn/docs/generated/tensorplay.conv1d_grad_bias.html tensorplay.conv1d_grad_bias tensorplay. conv1d_grad_bias ( grad_output , input , weight , stride , padding , dilation , groups ) [source] # tensorplay.conv1d_grad_input URL: https://www.tensorplay.cn/docs/generated/tensorplay.conv1d_grad_input.html tensorplay.conv1d_grad_input tensorplay. conv1d_grad_input ( grad_output , input , weight , stride , padding , dilation , groups ) [source] # tensorplay.conv1d_grad_weight URL: https://www.tensorplay.cn/docs/generated/tensorplay.conv1d_grad_weight.html tensorplay.conv1d_grad_weight tensorplay. conv1d_grad_weight ( grad_output , input , weight , stride , padding , dilation , groups ) [source] # tensorplay.conv2d_grad_bias URL: https://www.tensorplay.cn/docs/generated/tensorplay.conv2d_grad_bias.html tensorplay.conv2d_grad_bias tensorplay. conv2d_grad_bias ( grad_output , input , weight , stride , padding , dilation , groups ) [source] # tensorplay.conv2d_grad_input URL: https://www.tensorplay.cn/docs/generated/tensorplay.conv2d_grad_input.html tensorplay.conv2d_grad_input tensorplay. conv2d_grad_input ( grad_output , input , weight , stride , padding , dilation , groups ) [source] # tensorplay.conv2d_grad_weight URL: https://www.tensorplay.cn/docs/generated/tensorplay.conv2d_grad_weight.html tensorplay.conv2d_grad_weight tensorplay. conv2d_grad_weight ( grad_output , input , weight , stride , padding , dilation , groups ) [source] # tensorplay.conv2d_relu URL: https://www.tensorplay.cn/docs/generated/tensorplay.conv2d_relu.html tensorplay.conv2d_relu tensorplay. conv2d_relu ( input , weight , bias = None , stride = [1, 1] , padding = [0, 0] , dilation = [1, 1] , groups = 1 ) [source] # tensorplay.conv2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.conv2d.html tensorplay.conv2d tensorplay. conv2d ( input , weight , bias = None , stride = [1, 1] , padding = [0, 0] , dilation = [1, 1] , groups = 1 ) [source] # tensorplay.conv3d_grad_bias URL: https://www.tensorplay.cn/docs/generated/tensorplay.conv3d_grad_bias.html tensorplay.conv3d_grad_bias tensorplay. conv3d_grad_bias ( grad_output , input , weight , stride , padding , dilation , groups ) [source] # tensorplay.conv3d_grad_input URL: https://www.tensorplay.cn/docs/generated/tensorplay.conv3d_grad_input.html tensorplay.conv3d_grad_input tensorplay. conv3d_grad_input ( grad_output , input , weight , stride , padding , dilation , groups ) [source] # tensorplay.conv3d_grad_weight URL: https://www.tensorplay.cn/docs/generated/tensorplay.conv3d_grad_weight.html tensorplay.conv3d_grad_weight tensorplay. conv3d_grad_weight ( grad_output , input , weight , stride , padding , dilation , groups ) [source] # tensorplay.copy_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.copy_.html tensorplay.copy_ tensorplay. copy_ ( input , src , non_blocking = False ) [source] # tensorplay.cuda.AcceleratorError URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.AcceleratorError.html tensorplay.cuda.AcceleratorError exception tensorplay.cuda. AcceleratorError [source] # tensorplay.cuda.caching_allocator_alloc URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.caching_allocator_alloc.html tensorplay.cuda.caching_allocator_alloc tensorplay.cuda. caching_allocator_alloc ( size , device : Any | None = None , stream = None ) [source] Perform a memory allocation using the CUDA memory allocator. Not exposed by this TensorPlay build. # tensorplay.cuda.caching_allocator_delete URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.caching_allocator_delete.html tensorplay.cuda.caching_allocator_delete tensorplay.cuda. caching_allocator_delete ( mem_ptr ) [source] Delete memory allocated using the CUDA memory allocator. # tensorplay.cuda.can_device_access_peer URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.can_device_access_peer.html tensorplay.cuda.can_device_access_peer tensorplay.cuda. can_device_access_peer ( device : Any , peer_device : Any ) → bool [source] Check if peer access between two devices is possible. # tensorplay.cuda.change_current_allocator URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.change_current_allocator.html tensorplay.cuda.change_current_allocator tensorplay.cuda. change_current_allocator ( alloc ) [source] Change the currently used memory allocator (not supported by this build). # tensorplay.cuda.check_error URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.check_error.html tensorplay.cuda.check_error tensorplay.cuda. check_error ( res : int ) → None [source] Raise an error if the result of a CUDA runtime API call is not success. # tensorplay.cuda.classproperty URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.classproperty.html tensorplay.cuda.classproperty tensorplay.cuda. classproperty ( func : Callable ) → _ClassPropertyDescriptor [source] # tensorplay.cuda.clock_rate URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.clock_rate.html tensorplay.cuda.clock_rate tensorplay.cuda. clock_rate ( device : Any | None = None ) → int [source] Return the clock speed of the GPU SM in MHz (megahertz) over the past sample period as given by nvidia-smi . # tensorplay.cuda.CudaError URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.CudaError.html tensorplay.cuda.CudaError exception tensorplay.cuda. CudaError ( code : int ) [source] # CUDAPluggableAllocator URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.CUDAPluggableAllocator.html CUDAPluggableAllocator class tensorplay.cuda. CUDAPluggableAllocator ( so_file : str , alloc_fn_name : str = 'my_alloc' , free_fn_name : str = 'my_free' ) [source] CUDA memory allocator plugin (not supported by this TensorPlay build). # tensorplay.cuda.cudart URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.cudart.html tensorplay.cuda.cudart tensorplay.cuda. cudart ( ) [source] Retrieves the CUDA runtime API module. This function initializes the CUDA runtime environment if it is not already initialized and returns the CUDA runtime API module (_cudart). Returns : The CUDA runtime API module, or None when no ctypes runtime binding is exposed by this build. Return type : module or None # tensorplay.cuda.current_blas_handle URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.current_blas_handle.html tensorplay.cuda.current_blas_handle tensorplay.cuda. current_blas_handle ( ) [source] Return cublasHandle_t pointer to current cuBLAS handle # tensorplay.cuda.current_device URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.current_device.html tensorplay.cuda.current_device tensorplay.cuda. current_device ( ) → int [source] Return the index of a currently selected device. # tensorplay.cuda.current_solver_handle URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.current_solver_handle.html tensorplay.cuda.current_solver_handle tensorplay.cuda. current_solver_handle ( ) [source] Return cusolverDnHandle_t pointer to current cuSOLVER handle # tensorplay.cuda.current_stream URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.current_stream.html tensorplay.cuda.current_stream tensorplay.cuda. current_stream ( device : Any | None = None ) → Stream [source] Return the currently selected Stream for a given device. Parameters : device ( tensorplay.Device or int , optional ) – selected device. Returns the currently selected Stream for the current device, given by current_device() , if device is None (default). # tensorplay.cuda.default_stream URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.default_stream.html tensorplay.cuda.default_stream tensorplay.cuda. default_stream ( device : Any | None = None ) → Stream [source] Return the default Stream for a given device. Parameters : device ( tensorplay.Device or int , optional ) – selected device. Returns the default Stream for the current device, given by current_device() , if device is None (default). # tensorplay.cuda.DeferredCudaCallError URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.DeferredCudaCallError.html tensorplay.cuda.DeferredCudaCallError exception tensorplay.cuda. DeferredCudaCallError [source] # tensorplay.cuda.device_count URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.device_count.html tensorplay.cuda.device_count tensorplay.cuda. device_count ( ) → int [source] Return the number of GPUs available. Note This API will NOT poison fork if NVML discovery succeeds. # tensorplay.cuda.device_memory_used URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.device_memory_used.html tensorplay.cuda.device_memory_used tensorplay.cuda. device_memory_used ( device : Any | None = None ) → int [source] Return used global (device) memory in bytes as given by nvidia-smi . Parameters : device ( tensorplay.Device or int , optional ) – selected device. Returns statistic for the current device, given by current_device() , if device is None (default). # device_of URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.device_of.html device_of class tensorplay.cuda. device_of ( obj ) [source] Context-manager that changes the current device to that of given object. You can use both tensors and storages as arguments. If a given object is not allocated on a GPU, this is a no-op. Parameters : obj ( Tensor or Storage ) – object allocated on the selected device. # device URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.device.html device class tensorplay.cuda. device ( device : Any ) [source] Context-manager that changes the selected device. Parameters : device ( tensorplay.Device or int ) – device index to select. It’s a no-op if this argument is a negative integer or None . # Device URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.Device.html Device class tensorplay.cuda. Device is_cpu ( self : tensorplay._C.Device ) → bool is_cuda ( self : tensorplay._C.Device ) → bool # tensorplay.cuda.empty_cache URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.empty_cache.html tensorplay.cuda.empty_cache tensorplay.cuda. empty_cache ( ) → None [source] Release all unoccupied cached memory currently held by the caching allocator so that those can be used in other GPU application and visible in nvidia-smi . Note empty_cache() doesn’t increase the amount of GPU memory available for TensorPlay. However, it may help reduce fragmentation of GPU memory in certain cases. # GdsFile URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.gds.GdsFile.html GdsFile class tensorplay.cuda.gds. GdsFile ( * args , ** kwargs ) [source] A GDS file handle (not supported by this TensorPlay build). # tensorplay.cuda.get_allocator_backend URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.get_allocator_backend.html tensorplay.cuda.get_allocator_backend tensorplay.cuda. get_allocator_backend ( ) → str [source] Returns the active allocator backend as a string. Always native here. # tensorplay.cuda.get_arch_list URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.get_arch_list.html tensorplay.cuda.get_arch_list tensorplay.cuda. get_arch_list ( ) → list [ str ] [source] Return list CUDA architectures this library was compiled for. # tensorplay.cuda.get_device_capability URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.get_device_capability.html tensorplay.cuda.get_device_capability tensorplay.cuda. get_device_capability ( device : Any | None = None ) → tuple [ int , int ] [source] Get the cuda capability of a device. Parameters : device ( tensorplay.Device or int or str , optional ) – device for which to return the device capability. This function is a no-op if this argument is a negative integer. It uses the current device, given by current_device() , if device is None (default). Returns : the major and minor cuda capability of the device Return type : tuple ( int , int ) # tensorplay.cuda.get_device_name URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.get_device_name.html tensorplay.cuda.get_device_name tensorplay.cuda. get_device_name ( device : Any | None = None ) → str [source] Get the name of a device. Parameters : device ( tensorplay.Device or int or str , optional ) – device for which to return the name. This function is a no-op if this argument is a negative integer. It uses the current device, given by current_device() , if device is None (default). Returns : the name of the device Return type : str # tensorplay.cuda.get_device_properties URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.get_device_properties.html tensorplay.cuda.get_device_properties tensorplay.cuda. get_device_properties ( device : Any | None = None ) → _CudaDeviceProperties [source] Get the properties of a device. Parameters : device ( tensorplay.Device or int or str , optional ) – device for which to return the properties of the device. It uses the current device, given by current_device() , if device is None (default). Returns : the properties of the device Return type : _CudaDeviceProperties # tensorplay.cuda.get_gencode_flags URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.get_gencode_flags.html tensorplay.cuda.get_gencode_flags tensorplay.cuda. get_gencode_flags ( ) → str [source] Return NVCC gencode flags this library was compiled with. # tensorplay.cuda.get_per_process_memory_fraction URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.get_per_process_memory_fraction.html tensorplay.cuda.get_per_process_memory_fraction tensorplay.cuda. get_per_process_memory_fraction ( device : Any | None = None ) → float [source] Get memory fraction for a process. Always returns 1.0 in this build. # tensorplay.cuda.get_stream_from_external URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.get_stream_from_external.html tensorplay.cuda.get_stream_from_external tensorplay.cuda. get_stream_from_external ( data_ptr : int , device : Any | None = None ) → Stream [source] Return a Stream from an externally allocated CUDA stream. # tensorplay.cuda.get_sync_debug_mode URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.get_sync_debug_mode.html tensorplay.cuda.get_sync_debug_mode tensorplay.cuda. get_sync_debug_mode ( ) → int [source] Return current value of debug mode for cuda synchronizing operations. Always returns 0 in this build (mode is not enforced). # tensorplay.cuda.graph_annotations.clear_kernel_annotations URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.graph_annotations.clear_kernel_annotations.html tensorplay.cuda.graph_annotations.clear_kernel_annotations tensorplay.cuda.graph_annotations. clear_kernel_annotations ( ) [source] Clear all recorded kernel annotations. # tensorplay.cuda.graph_annotations.get_kernel_annotations URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.graph_annotations.get_kernel_annotations.html tensorplay.cuda.graph_annotations.get_kernel_annotations tensorplay.cuda.graph_annotations. get_kernel_annotations ( ) [source] Return the recorded kernel annotations mapping. # tensorplay.cuda.graph_annotations.is_available URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.graph_annotations.is_available.html tensorplay.cuda.graph_annotations.is_available tensorplay.cuda.graph_annotations. is_available ( ) → bool [source] Whether graph-kernel annotation recording is supported (always False here). # tensorplay.cuda.graph_annotations.mark_kernels URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.graph_annotations.mark_kernels.html tensorplay.cuda.graph_annotations.mark_kernels tensorplay.cuda.graph_annotations. mark_kernels ( name , metadata = None ) [source] Tag the GPU work captured within its scope with user metadata. # CUDAGraph URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.graphs.CUDAGraph.html CUDAGraph class tensorplay.cuda.graphs. CUDAGraph [source] Wrapper around a CUDA graph, mirroring torch.cuda.CUDAGraph . instantiate ( ) [source] Compile the captured template (happens automatically at replay). reset ( ) [source] Destroy the executable and free its private memory pool. All tensors allocated during the capture must be released first. # tensorplay.cuda.graphs.export_dot URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.graphs.export_dot.html tensorplay.cuda.graphs.export_dot tensorplay.cuda.graphs. export_dot ( file_path : str ) → str [source] Export the last captured CUDA graph to a DOT file. # tensorplay.cuda.graphs.export_graph_data URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.graphs.export_graph_data.html tensorplay.cuda.graphs.export_graph_data tensorplay.cuda.graphs. export_graph_data ( graph_id : int ) → dict [source] Serialize a captured CUDA graph into a dictionary of node data. # tensorplay.cuda.graphs.graph_pool_handle URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.graphs.graph_pool_handle.html tensorplay.cuda.graphs.graph_pool_handle tensorplay.cuda.graphs. graph_pool_handle ( ) [source] Return an opaque token representing the id of a graph memory pool. # tensorplay.cuda.graphs.graph URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.graphs.graph.html tensorplay.cuda.graphs.graph tensorplay.cuda.graphs. graph ( cuda_graph , pool = None , stream = None , capture_error_mode = 'global' ) [source] Context-manager that captures CUDA work into a tensorplay.cuda.CUDAGraph . Parameters : cuda_graph ( CUDAGraph ) – the graph object to capture into. pool – unsupported (each capture gets its own private memory pool). stream ( Stream , optional ) – unsupported; capture always runs on the dedicated side stream exposed by the runtime. capture_error_mode ( str , optional ) – "global" (default) makes any unsafe CUDA call anywhere in the process fail the capture. "thread_local" and "relaxed" are rejected because the native binding always captures in global mode. # tensorplay.cuda.graphs.is_current_stream_capturing URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.graphs.is_current_stream_capturing.html tensorplay.cuda.graphs.is_current_stream_capturing tensorplay.cuda.graphs. is_current_stream_capturing ( ) [source] Return True if CUDA graph capture is underway on the current thread. # tensorplay.cuda.graphs.make_graphed_callables URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.graphs.make_graphed_callables.html tensorplay.cuda.graphs.make_graphed_callables tensorplay.cuda.graphs. make_graphed_callables ( callables , sample_args , num_warmup_iters = 3 , allow_unused_input = False ) [source] Callables that run per-iteration with CUDA graph capture (not supported). # GreenContext URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.green_contexts.GreenContext.html GreenContext class tensorplay.cuda.green_contexts. GreenContext ( * args , ** kwargs ) # tensorplay.cuda.host_memory_stats_as_nested_dict URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.host_memory_stats_as_nested_dict.html tensorplay.cuda.host_memory_stats_as_nested_dict tensorplay.cuda. host_memory_stats_as_nested_dict ( ) → dict [ str , Any ] [source] Return the result of host_memory_stats() as a nested dictionary. # tensorplay.cuda.host_memory_stats URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.host_memory_stats.html tensorplay.cuda.host_memory_stats tensorplay.cuda. host_memory_stats ( ) → dict [ str , Any ] [source] Return a dictionary of pinned (host) allocator statistics. # tensorplay.cuda.init URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.init.html tensorplay.cuda.init tensorplay.cuda. init ( ) [source] Initialize TensorPlay’s CUDA state. You may need to call this explicitly if you are interacting with TensorPlay via its C API, as Python bindings for CUDA functionality will not be available until this initialization takes place. Ordinary users should not need this, as all of TensorPlay’s CUDA methods automatically initialize CUDA state on-demand. Does nothing if the CUDA state is already initialized. # tensorplay.cuda.ipc_collect URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.ipc_collect.html tensorplay.cuda.ipc_collect tensorplay.cuda. ipc_collect ( ) [source] Force collects GPU memory after it has been released by CUDA IPC. # tensorplay.cuda.is_available URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.is_available.html tensorplay.cuda.is_available tensorplay.cuda. is_available ( ) → bool [source] Return a bool indicating if CUDA is currently available. Note This function will NOT poison fork if the environment variable TENSORPLAY_NVML_BASED_CUDA_CHECK=1 is set. # tensorplay.cuda.is_bf16_supported URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.is_bf16_supported.html tensorplay.cuda.is_bf16_supported tensorplay.cuda. is_bf16_supported ( including_emulation : bool = True ) [source] Return a bool indicating if the current CUDA device supports dtype bfloat16. # tensorplay.cuda.is_gds_available URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.is_gds_available.html tensorplay.cuda.is_gds_available tensorplay.cuda. is_gds_available ( ) → bool [source] Return whether GDS is available. Always False in this build. # tensorplay.cuda.is_initialized URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.is_initialized.html tensorplay.cuda.is_initialized tensorplay.cuda. is_initialized ( ) [source] Return whether TensorPlay’s CUDA state has been initialized. # tensorplay.cuda.is_tf32_supported URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.is_tf32_supported.html tensorplay.cuda.is_tf32_supported tensorplay.cuda. is_tf32_supported ( ) → bool [source] Return a bool indicating if the current CUDA device supports dtype tf32. # tensorplay.cuda.list_gpu_processes URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.list_gpu_processes.html tensorplay.cuda.list_gpu_processes tensorplay.cuda. list_gpu_processes ( device : Any | None = None ) → str [source] Return a human-readable printout of the running processes and their GPU memory use for a given device. # tensorplay.cuda.max_memory_allocated URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.max_memory_allocated.html tensorplay.cuda.max_memory_allocated tensorplay.cuda. max_memory_allocated ( device : Any | None = None ) → int [source] Return the maximum GPU memory occupied by tensors in bytes for a given device. By default, this returns the peak allocated memory since the beginning of this program. reset_peak_memory_stats() can be used to reset the starting point in tracking this metric. Parameters : device ( tensorplay.Device or int , optional ) – selected device. Returns statistic for the current device, given by current_device() , if device is None (default). # tensorplay.cuda.max_memory_cached URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.max_memory_cached.html tensorplay.cuda.max_memory_cached tensorplay.cuda. max_memory_cached ( device : Any | None = None ) → int [source] Deprecated; see max_memory_reserved() . # tensorplay.cuda.max_memory_reserved URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.max_memory_reserved.html tensorplay.cuda.max_memory_reserved tensorplay.cuda. max_memory_reserved ( device : Any | None = None ) → int [source] Return the maximum GPU memory managed by the caching allocator in bytes for a given device. Parameters : device ( tensorplay.Device or int , optional ) – selected device. Returns statistic for the current device, given by current_device() , if device is None (default). # tensorplay.cuda.mem_get_info URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.mem_get_info.html tensorplay.cuda.mem_get_info tensorplay.cuda. mem_get_info ( device : Any | None = None ) → tuple [ int , int ] [source] Return the global free and total GPU memory for a given device using cudaMemGetInfo. # tensorplay.cuda.memory_allocated URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.memory_allocated.html tensorplay.cuda.memory_allocated tensorplay.cuda. memory_allocated ( device : Any | None = None ) → int [source] Return the current GPU memory occupied by tensors in bytes for a given device. Parameters : device ( tensorplay.Device or int , optional ) – selected device. Returns statistic for the current device, given by current_device() , if device is None (default). # tensorplay.cuda.memory_cached URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.memory_cached.html tensorplay.cuda.memory_cached tensorplay.cuda. memory_cached ( device : Any | None = None ) → int [source] Deprecated; see memory_reserved() . # tensorplay.cuda.memory_reserved URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.memory_reserved.html tensorplay.cuda.memory_reserved tensorplay.cuda. memory_reserved ( device : Any | None = None ) → int [source] Return the current GPU memory managed by the caching allocator in bytes for a given device. Parameters : device ( tensorplay.Device or int , optional ) – selected device. Returns statistic for the current device, given by current_device() , if device is None (default). # tensorplay.cuda.memory_snapshot URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.memory_snapshot.html tensorplay.cuda.memory_snapshot tensorplay.cuda. memory_snapshot ( mempool_id = None , include_traces = True ) [source] Return a snapshot of the CUDA memory allocator state across all devices. Interpreting the output of this function requires familiarity with the memory allocator internals. Not exposed by this TensorPlay build. # tensorplay.cuda.memory_stats_as_nested_dict URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.memory_stats_as_nested_dict.html tensorplay.cuda.memory_stats_as_nested_dict tensorplay.cuda. memory_stats_as_nested_dict ( device : Any | None = None ) → dict [ str , Any ] [source] Return the result of memory_stats() as a nested dictionary. # tensorplay.cuda.memory_stats URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.memory_stats.html tensorplay.cuda.memory_stats tensorplay.cuda. memory_stats ( device : Any | None = None ) → dict [ str , Any ] [source] Return a dictionary of CUDA memory allocator statistics for a given device. The return value of this function is a dictionary of statistics, each of which is a non-negative integer. See torch.cuda.memory_stats() for the full key layout; keys that are not tracked by this TensorPlay build are always reported as zero. Parameters : device ( tensorplay.Device or int , optional ) – selected device. Returns statistics for the current device, given by current_device() , if device is None (default). # tensorplay.cuda.memory_summary URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.memory_summary.html tensorplay.cuda.memory_summary tensorplay.cuda. memory_summary ( device : Any | None = None , abbreviated : bool = False ) → str [source] Return a human-readable printout of the current memory allocator statistics for a given device. This can be useful to display periodically during training, or when handling out-of-memory exceptions. Parameters : device ( tensorplay.Device or int , optional ) – selected device. Returns printout for the current device, given by current_device() , if device is None (default). abbreviated ( bool , optional ) – whether to return an abbreviated summary (default: False). # tensorplay.cuda.memory_usage URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.memory_usage.html tensorplay.cuda.memory_usage tensorplay.cuda. memory_usage ( device : Any | None = None ) → int [source] Return the percent of time over the past sample period during which global (device) memory was being read or written as given by nvidia-smi . Warning: Each sample period may be between 1 second and 1/6 second, depending on the product being queried. # tensorplay.cuda.memory.caching_allocator_disabled URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.memory.caching_allocator_disabled.html tensorplay.cuda.memory.caching_allocator_disabled tensorplay.cuda.memory. caching_allocator_disabled ( ) [source] Context manager that temporarily disables the CUDA caching allocator. # tensorplay.cuda.memory.caching_allocator_enable URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.memory.caching_allocator_enable.html tensorplay.cuda.memory.caching_allocator_enable tensorplay.cuda.memory. caching_allocator_enable ( value : bool = True ) → None [source] Enable or disable the CUDA memory allocator. On by default. # tensorplay.cuda.memory.use_mem_pool URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.memory.use_mem_pool.html tensorplay.cuda.memory.use_mem_pool tensorplay.cuda.memory. use_mem_pool ( pool ) [source] Route allocations to a MemPool (not supported by this build). # MemPool URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.MemPool.html MemPool class tensorplay.cuda. MemPool ( * args , ** kwargs ) [source] MemPool context (not supported by this TensorPlay build). # tensorplay.cuda.nccl.is_available URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.nccl.is_available.html tensorplay.cuda.nccl.is_available tensorplay.cuda.nccl. is_available ( tensors ) → bool [source] # tensorplay.cuda.nccl.version URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.nccl.version.html tensorplay.cuda.nccl.version tensorplay.cuda.nccl. version ( ) [source] Returns the version of the NCCL. This function returns a tuple containing the major, minor, and patch version numbers of the NCCL. The suffix is also included in the tuple if a version suffix exists. :returns: The version information of the NCCL. :rtype: tuple # tensorplay.cuda.nvtx.mark URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.nvtx.mark.html tensorplay.cuda.nvtx.mark tensorplay.cuda.nvtx. mark ( msg ) [source] Describe an instantaneous event that occurred at some point. Parameters : msg ( str ) – ASCII message to associate with the event. # tensorplay.cuda.nvtx.range_end URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.nvtx.range_end.html tensorplay.cuda.nvtx.range_end tensorplay.cuda.nvtx. range_end ( range_id ) → None [source] Mark the end of a range for a given range_id. Parameters : range_id ( int ) – a unique handle for the start range. # tensorplay.cuda.nvtx.range_pop URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.nvtx.range_pop.html tensorplay.cuda.nvtx.range_pop tensorplay.cuda.nvtx. range_pop ( ) [source] Pop a range off of a stack of nested range spans. Returns the zero-based depth of the range that is ended. # tensorplay.cuda.nvtx.range_push URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.nvtx.range_push.html tensorplay.cuda.nvtx.range_push tensorplay.cuda.nvtx. range_push ( msg ) [source] Push a range onto a stack of nested range span. Returns zero-based depth of the range that is started. Parameters : msg ( str ) – ASCII message to associate with range # tensorplay.cuda.nvtx.range_start URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.nvtx.range_start.html tensorplay.cuda.nvtx.range_start tensorplay.cuda.nvtx. range_start ( msg ) → int [source] Mark the start of a range with string message. It returns a unique handle for this range to pass to the corresponding call to rangeEnd(). Returns: A range handle (uint64_t) that can be passed to range_end(). Parameters : msg ( str ) – ASCII message to associate with the range. # tensorplay.cuda.nvtx.range URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.nvtx.range.html tensorplay.cuda.nvtx.range tensorplay.cuda.nvtx. range ( msg , * args , ** kwargs ) [source] Context manager / decorator that pushes an NVTX range at the beginning of its scope, and pops it at the end. If extra arguments are given, they are passed as arguments to msg.format(). Parameters : msg ( str ) – message to associate with the range # tensorplay.cuda.OutOfMemoryError URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.OutOfMemoryError.html tensorplay.cuda.OutOfMemoryError exception tensorplay.cuda. OutOfMemoryError [source] # tensorplay.cuda.power_draw URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.power_draw.html tensorplay.cuda.power_draw tensorplay.cuda. power_draw ( device : Any | None = None ) → int [source] Return the average power draw of the GPU sensor in mW (MilliWatts) over the past sample period as given by nvidia-smi for Fermi or newer fully supported devices. # tensorplay.cuda.profiler.profile URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.profiler.profile.html tensorplay.cuda.profiler.profile tensorplay.cuda.profiler. profile ( ) [source] Enable profiling. Context Manager to enabling profile collection by the active profiling tool from CUDA backend. .. rubric:: Example >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_CUDA) >>> import tensorplay as tp >>> model = tp . nn . Linear ( 20 , 30 ) . cuda () >>> inputs = tp . randn ( 128 , 20 ) . cuda () >>> with tp . cuda . profiler . profile (): ... model ( inputs ) # tensorplay.cuda.profiler.start URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.profiler.start.html tensorplay.cuda.profiler.start tensorplay.cuda.profiler. start ( ) [source] Starts cuda profiler data collection. Warning Raises CudaError in case of it is unable to start the profiler. # tensorplay.cuda.profiler.stop URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.profiler.stop.html tensorplay.cuda.profiler.stop tensorplay.cuda.profiler. stop ( ) [source] Stops cuda profiler data collection. Warning Raises CudaError in case of it is unable to stop the profiler. # tensorplay.cuda.random.get_rng_state_all URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.random.get_rng_state_all.html tensorplay.cuda.random.get_rng_state_all tensorplay.cuda.random. get_rng_state_all ( ) → list [ TensorBase ] [source] Return a list of ByteTensor representing the random number states of all devices. # tensorplay.cuda.random.get_rng_state URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.random.get_rng_state.html tensorplay.cuda.random.get_rng_state tensorplay.cuda.random. get_rng_state ( device : int | str | Any = 'cuda' ) → TensorBase [source] Return the random number generator state of the specified GPU as a ByteTensor. Parameters : device ( tensorplay.Device or int , optional ) – The device to return the RNG state of. Default: 'cuda' (i.e., the current CUDA device). Warning This function eagerly initializes CUDA. # tensorplay.cuda.random.initial_seed URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.random.initial_seed.html tensorplay.cuda.random.initial_seed tensorplay.cuda.random. initial_seed ( ) → int [source] Return the current random seed of the current GPU. Warning This function eagerly initializes CUDA. # tensorplay.cuda.random.manual_seed_all URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.random.manual_seed_all.html tensorplay.cuda.random.manual_seed_all tensorplay.cuda.random. manual_seed_all ( seed : int ) → None [source] Set the seed for generating random numbers on all GPUs. It’s safe to call this function if CUDA is not available; in that case, it is silently ignored. Parameters : seed ( int ) – The desired seed. # tensorplay.cuda.random.manual_seed URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.random.manual_seed.html tensorplay.cuda.random.manual_seed tensorplay.cuda.random. manual_seed ( seed : int ) → None [source] Set the seed for generating random numbers for the current GPU. It’s safe to call this function if CUDA is not available; in that case, it is silently ignored. Parameters : seed ( int ) – The desired seed. Warning If you are working with a multi-GPU model, this function is insufficient to get determinism. To seed all GPUs, use manual_seed_all() . # tensorplay.cuda.random.seed_all URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.random.seed_all.html tensorplay.cuda.random.seed_all tensorplay.cuda.random. seed_all ( ) → None [source] Set the seed for generating random numbers to a random number on all GPUs. It’s safe to call this function if CUDA is not available; in that case, it is silently ignored. # tensorplay.cuda.random.seed URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.random.seed.html tensorplay.cuda.random.seed tensorplay.cuda.random. seed ( ) → None [source] Set the seed for generating random numbers to a random number for the current GPU. It’s safe to call this function if CUDA is not available; in that case, it is silently ignored. Warning If you are working with a multi-GPU model, this function will only initialize the seed on one GPU. To initialize all GPUs, use seed_all() . # tensorplay.cuda.random.set_rng_state_all URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.random.set_rng_state_all.html tensorplay.cuda.random.set_rng_state_all tensorplay.cuda.random. set_rng_state_all ( new_states : Iterable [ TensorBase ] ) → None [source] Set the random number generator state of all devices. Parameters : new_states ( Iterable of ByteTensor ) – The desired state for each device. # tensorplay.cuda.random.set_rng_state URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.random.set_rng_state.html tensorplay.cuda.random.set_rng_state tensorplay.cuda.random. set_rng_state ( new_state : TensorBase , device : int | str | Any = 'cuda' ) → None [source] Set the random number generator state of the specified GPU. Parameters : new_state ( ByteTensor ) – The desired state device ( tensorplay.Device or int , optional ) – The device to set the RNG state. Default: 'cuda' (i.e., the current CUDA device). # tensorplay.cuda.reset_accumulated_host_memory_stats URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.reset_accumulated_host_memory_stats.html tensorplay.cuda.reset_accumulated_host_memory_stats tensorplay.cuda. reset_accumulated_host_memory_stats ( ) → None [source] Reset the “accumulated” (historical) stats tracked by the host memory allocator. # tensorplay.cuda.reset_accumulated_memory_stats URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.reset_accumulated_memory_stats.html tensorplay.cuda.reset_accumulated_memory_stats tensorplay.cuda. reset_accumulated_memory_stats ( device : Any | None = None ) → None [source] Reset the “accumulated” (historical) stats tracked by the CUDA memory allocator. This TensorPlay build does not track historical totals; the call is a no-op. # tensorplay.cuda.reset_max_memory_allocated URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.reset_max_memory_allocated.html tensorplay.cuda.reset_max_memory_allocated tensorplay.cuda. reset_max_memory_allocated ( device : Any | None = None ) → None [source] Reset the starting point in tracking maximum GPU memory occupied by tensors for a given device. Warning This function now calls reset_peak_memory_stats() , which resets /all/ peak memory stats. # tensorplay.cuda.reset_max_memory_cached URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.reset_max_memory_cached.html tensorplay.cuda.reset_max_memory_cached tensorplay.cuda. reset_max_memory_cached ( device : Any | None = None ) → None [source] Reset the starting point in tracking maximum GPU memory managed by the caching allocator for a given device. Warning This function now calls reset_peak_memory_stats() , which resets /all/ peak memory stats. # tensorplay.cuda.reset_peak_host_memory_stats URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.reset_peak_host_memory_stats.html tensorplay.cuda.reset_peak_host_memory_stats tensorplay.cuda. reset_peak_host_memory_stats ( ) → None [source] Reset the “peak” stats tracked by the host memory allocator. # tensorplay.cuda.reset_peak_memory_stats URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.reset_peak_memory_stats.html tensorplay.cuda.reset_peak_memory_stats tensorplay.cuda. reset_peak_memory_stats ( device : Any | None = None ) → None [source] Reset the “peak” stats tracked by the CUDA memory allocator. Peak stats correspond to the “peak” key in each individual stat dict. Parameters : device ( tensorplay.Device or int , optional ) – selected device. Returns statistic for the current device, given by current_device() , if device is None (default). # tensorplay.cuda.set_device URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.set_device.html tensorplay.cuda.set_device tensorplay.cuda. set_device ( device : Any ) → None [source] Set the current device. Usage of this function is discouraged in favor of device . In most cases it’s better to use CUDA_VISIBLE_DEVICES environmental variable. Parameters : device ( tensorplay.Device or int ) – selected device. This function is a no-op if this argument is negative. # tensorplay.cuda.set_per_process_memory_fraction URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.set_per_process_memory_fraction.html tensorplay.cuda.set_per_process_memory_fraction tensorplay.cuda. set_per_process_memory_fraction ( fraction , device : Any | None = None ) → None [source] Set memory fraction for a process. Not enforced by this TensorPlay build; validated for signature parity. Parameters : fraction ( float ) – Range: 0~1. Allowed memory equals total_memory * fraction. device ( tensorplay.Device or int , optional ) – selected device. # tensorplay.cuda.set_stream URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.set_stream.html tensorplay.cuda.set_stream tensorplay.cuda. set_stream ( stream : Stream ) [source] Set the current stream. This is a wrapper API to set the stream. Usage of this function is discouraged in favor of the stream context manager. Parameters : stream ( Stream ) – selected stream. This function is a no-op if this argument is None . # tensorplay.cuda.set_sync_debug_mode URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.set_sync_debug_mode.html tensorplay.cuda.set_sync_debug_mode tensorplay.cuda. set_sync_debug_mode ( debug_mode : int | str ) → None [source] Set the debug mode for cuda synchronizing operations. Not enforced by this TensorPlay build; validated for signature parity. Parameters : debug_mode ( str or int ) – if “default” or 0, don’t error or warn on synchronizing operations, if “warn” or 1, warn on synchronizing operations, if “error” or 2, error out synchronizing operations. # tensorplay.cuda.stream URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.stream.html tensorplay.cuda.stream tensorplay.cuda. stream ( stream_ : Stream | None ) → StreamContext [source] Wrap around the Context-manager StreamContext that selects a given stream. Parameters : stream ( Stream ) – selected stream. This manager is a no-op if it’s None . Note Streams are per-device. # StreamContext URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.StreamContext.html StreamContext class tensorplay.cuda. StreamContext ( stream : Stream | None ) [source] Context-manager that selects a given stream. All CUDA kernels queued within its context will be enqueued on a selected stream. Parameters : Stream ( Stream ) – selected stream. This manager is a no-op if it’s None . Note Streams are per-device. # Event URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.streams.Event.html Event class tensorplay.cuda.streams. Event ( enable_timing = False , blocking = False , interprocess = False , external = False ) [source] Wrapper around a CUDA event. CUDA events are synchronization markers that can be used to monitor the device’s progress, to accurately measure timing, and to synchronize CUDA streams. Parameters : enable_timing ( bool , optional ) – indicates if the event should measure time (default: False ) blocking ( bool , optional ) – if True , wait() will be blocking (default: False ) interprocess ( bool ) – if True , the event can be shared between processes (default: False ) elapsed_time ( end_event : Event ) [source] Return the time elapsed. Time reported in milliseconds after the event was recorded and before the end_event was recorded. Parameters : end_event ( Event ) – the end event. classmethod from_ipc_handle ( device , handle ) [source] Reconstruct an event from an IPC handle on the given device. ipc_handle ( ) [source] Return an IPC handle of this event. query ( ) [source] Check if all work currently captured by event has completed. record ( stream : Stream | None = None ) [source] Record the event in a given stream. Parameters : stream ( Stream , optional ) – Uses tensorplay.cuda.current_stream() if no stream is specified. synchronize ( ) → None [source] Wait for the event to complete. wait ( stream : Stream | None = None ) → None [source] Make all future work submitted to the given stream wait for this event. Parameters : stream ( Stream , optional ) – Uses tensorplay.cuda.current_stream() if no stream is specified. # ExternalStream URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.streams.ExternalStream.html ExternalStream class tensorplay.cuda.streams. ExternalStream ( stream_ptr , device = None , ** kwargs ) [source] Wrapper around an externally allocated CUDA stream. This class is used to wrap streams allocated in other libraries in order to facilitate data exchange and multi-library interactions. Note This class doesn’t manage the stream life-cycle, it is the user responsibility to keep the referenced stream alive while this class is being used. Parameters : stream_ptr ( int ) – Integer representation of the cudaStream_t value allocated externally. device ( tensorplay.Device or int , optional ) – the device where the stream was originally allocated. If device is specified incorrectly, subsequent launches using this stream may fail. query ( ) → bool Check if all the work submitted has been completed. record_event ( event : Event | None = None ) Record an event. Parameters : event ( Event , optional ) – event to record. If not given, a new one will be allocated. Returns : Recorded event. synchronize ( ) → None Wait for all the kernels in this stream to complete. wait_event ( event : Event | tensorplay.Event ) → None Make all future work submitted to the stream wait for an event. wait_stream ( stream : Stream ) → None Synchronize with another stream. # Stream URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.streams.Stream.html Stream class tensorplay.cuda.streams. Stream ( device = None , priority = 0 , ** kwargs ) [source] Wrapper around a CUDA stream. A CUDA stream is a linear sequence of execution that belongs to a specific device, independent from other streams. It supports with statement as a context manager to ensure the operators within the with block are running on the corresponding stream. See the CUDA semantics documentation for details. Parameters : device ( tensorplay.Device or int , optional ) – a device on which to allocate the stream. If device is None (default) or a negative integer, this will use the current device. priority ( int , optional ) – priority of the stream, which can be positive, 0, or negative. A lower number indicates a higher priority. By default, the priority is set to 0. query ( ) → bool [source] Check if all the work submitted has been completed. record_event ( event : Event | None = None ) [source] Record an event. Parameters : event ( Event , optional ) – event to record. If not given, a new one will be allocated. Returns : Recorded event. synchronize ( ) → None [source] Wait for all the kernels in this stream to complete. wait_event ( event : Event | tensorplay.Event ) → None [source] Make all future work submitted to the stream wait for an event. wait_stream ( stream : Stream ) → None [source] Synchronize with another stream. # tensorplay.cuda.synchronize URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.synchronize.html tensorplay.cuda.synchronize tensorplay.cuda. synchronize ( device : Any | None = None ) → None [source] Wait for all kernels in all streams on a CUDA device to complete. Parameters : device ( tensorplay.Device or int , optional ) – device for which to synchronize. It uses the current device, given by current_device() , if device is None (default). # tensorplay.cuda.temperature URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.temperature.html tensorplay.cuda.temperature tensorplay.cuda. temperature ( device : Any | None = None ) → int [source] Return the average temperature of the GPU sensor in Degrees C (Centigrades). The average temperature is computed based on past sample period as given by nvidia-smi . # tensorplay.cuda.utilization URL: https://www.tensorplay.cn/docs/generated/tensorplay.cuda.utilization.html tensorplay.cuda.utilization tensorplay.cuda. utilization ( device : Any | None = None ) → int [source] Return the percent of time over the past sample period during which one or more kernels was executing on the GPU as given by nvidia-smi . Warning: Each sample period may be between 1 second and 1/6 second, depending on the product being queried. # tensorplay.custom_bwd URL: https://www.tensorplay.cn/docs/generated/tensorplay.custom_bwd.html tensorplay.custom_bwd tensorplay. custom_bwd ( bwd = None , * , device_type : str ) [source] Create a helper decorator for backward methods of custom autograd functions. Autograd functions are subclasses of tensorplay.autograd.Function . Ensures that backward executes with the same autocast state as forward . Parameters : device_type ( str ) – Device type to use. ‘cuda’, ‘cpu’. The type is the same as the type attribute of a tensorplay.device . Thus, you may obtain the device type of a tensor using Tensor.device.type . # tensorplay.custom_fwd URL: https://www.tensorplay.cn/docs/generated/tensorplay.custom_fwd.html tensorplay.custom_fwd tensorplay. custom_fwd ( fwd = None , * , device_type : str , cast_inputs = None ) [source] Create a helper decorator for forward methods of custom autograd functions. Autograd functions are subclasses of tensorplay.autograd.Function . Parameters : device_type ( str ) – Device type to use. ‘cuda’, ‘cpu’. The type is the same as the type attribute of a tensorplay.device . Thus, you may obtain the device type of a tensor using Tensor.device.type . cast_inputs ( tensorplay.dtype or None, optional, default=None) – If not None , when forward runs in an autocast-enabled region, casts incoming floating-point Tensors to the target dtype (non-floating-point Tensors are not affected), then executes forward with autocast disabled. If None , forward ’s internal ops execute with the current autocast state. Note If the decorated forward is called outside an autocast-enabled region, custom_fwd is a no-op and cast_inputs has no effect. # tensorplay.default_generator URL: https://www.tensorplay.cn/docs/generated/tensorplay.default_generator.html tensorplay.default_generator tensorplay. default_generator ( ) → tensorplay._C.Generator # tensorplay.dequantize_per_channel URL: https://www.tensorplay.cn/docs/generated/tensorplay.dequantize_per_channel.html tensorplay.dequantize_per_channel tensorplay. dequantize_per_channel ( input , scales , zero_points , axis = 0 ) [source] # tensorplay.dequantize_per_tensor URL: https://www.tensorplay.cn/docs/generated/tensorplay.dequantize_per_tensor.html tensorplay.dequantize_per_tensor tensorplay. dequantize_per_tensor ( input , scale , zero_point ) [source] # device URL: https://www.tensorplay.cn/docs/generated/tensorplay.device.html device tensorplay. device alias of Device # Device URL: https://www.tensorplay.cn/docs/generated/tensorplay.Device.html Device class tensorplay. Device is_cpu ( self : tensorplay._C.Device ) → bool is_cuda ( self : tensorplay._C.Device ) → bool # DeviceType URL: https://www.tensorplay.cn/docs/generated/tensorplay.DeviceType.html DeviceType class tensorplay. DeviceType Members: CPU CUDA property name # tensorplay.diagonal URL: https://www.tensorplay.cn/docs/generated/tensorplay.diagonal.html tensorplay.diagonal tensorplay. diagonal ( input , offset = 0 , dim1 = 0 , dim2 = 1 ) [source] # FileStore URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed._store.FileStore.html FileStore class tensorplay.distributed._store. FileStore ( file_name : str , world_size : int = -1 ) [source] Flock-based append-log store in a single file. # Store URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed._store.Store.html Store class tensorplay.distributed._store. Store [source] # TCPStore URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed._store.TCPStore.html TCPStore class tensorplay.distributed._store. TCPStore ( host_name : str , port : int = 0 , world_size : int = -1 , is_master : bool = False , timeout : float = 300.0 , wait_for_workers : bool = True ) [source] Client for a threaded TCP key-value server ( torch.distributed.TCPStore subset: set/get/add/delete/check as used by rendezvous and barriers). # tensorplay.distributed.algorithms.model_averaging.utils.average_parameters_or_parameter_groups URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.algorithms.model_averaging.utils.average_parameters_or_parameter_groups.html tensorplay.distributed.algorithms.model_averaging.utils.average_parameters_or_parameter_groups tensorplay.distributed.algorithms.model_averaging.utils. average_parameters_or_parameter_groups ( params : Iterable [ Parameter ] | Iterable [ dict [ str , Parameter ] ] , process_group : ProcessGroup ) [source] Averages parameters of a model or parameter groups of an optimizer. # tensorplay.distributed.algorithms.model_averaging.utils.average_parameters URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.algorithms.model_averaging.utils.average_parameters.html tensorplay.distributed.algorithms.model_averaging.utils.average_parameters tensorplay.distributed.algorithms.model_averaging.utils. average_parameters ( params : Iterator [ Parameter ] , process_group : ProcessGroup ) [source] Averages all the given parameters. For allreduce efficiency, all the parameters are flattened into a contiguous buffer. Thus, it requires extra memory of the same size as the given parameters. # tensorplay.distributed.algorithms.model_averaging.utils.get_params_to_average URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.algorithms.model_averaging.utils.get_params_to_average.html tensorplay.distributed.algorithms.model_averaging.utils.get_params_to_average tensorplay.distributed.algorithms.model_averaging.utils. get_params_to_average ( params : Iterable [ Parameter ] | Iterable [ dict [ str , Parameter ] ] ) [source] Return a list of parameters that need to average. This filters out the parameters that do not contain any gradients. :param params: The parameters of a model or parameter groups of an optimizer. # tensorplay.distributed.collective_utils.all_gather_object_enforce_type URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.collective_utils.all_gather_object_enforce_type.html tensorplay.distributed.collective_utils.all_gather_object_enforce_type tensorplay.distributed.collective_utils. all_gather_object_enforce_type ( pg: dist.ProcessGroup, object_list: list[Any], obj: Any, type_checker: Callable[[Any, Any], bool] = > ) → None [source] Similar to plain all_gather_object but with additional type checking AFTER gather is done to ensure basic consistency. If check does not pass, all ranks will fail with exception. This is generally to prevent conditional logic leading to unexpected messages being received. This is considered fatal code error, but due to logic stacks this might happen implicitly in practice. The default check does not check sub type (considered different) or covariance (considered same) but users can pass in custom checker if more complicated check is needed. # DeviceMesh URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.device_mesh.DeviceMesh.html DeviceMesh class tensorplay.distributed.device_mesh. DeviceMesh ( device_type : str , mesh = None , * , mesh_dim_names = None , _dim_group_names = None , _rank_map = None , _sizes = None , _strides = None , _root_mesh = None ) [source] DeviceMesh represents a mesh of devices (torch parity). The mesh is an n-d array whose values are global ranks. Process groups are created per mesh dimension so collectives can run on each dimension independently. Example: >>> from tensorplay.distributed.device_mesh import init_device_mesh >>> mesh = init_device_mesh ( "cuda" , mesh_shape = ( 2 , 4 ), ... mesh_dim_names = ( "dp" , "tp" )) classmethod from_group ( group , device_type = None , mesh = None , mesh_dim_names = None ) → DeviceMesh [source] Construct a 1-D DeviceMesh from an existing ProcessGroup. get_coordinate ( ) → tuple [ int , ... ] | None [source] Returns this rank’s coordinate in the mesh, or None if absent. get_group ( mesh_dim = None ) [source] Returns the process group along mesh_dim (torch parity). get_local_rank ( mesh_dim = None ) → int [source] Returns local rank along mesh_dim (torch parity). # tensorplay.distributed.device_mesh.init_device_mesh URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.device_mesh.init_device_mesh.html tensorplay.distributed.device_mesh.init_device_mesh tensorplay.distributed.device_mesh. init_device_mesh ( device_type : str , mesh_shape : tuple [ int , ... ] , * , mesh_dim_names : tuple [ str , ... ] | None = None , backend_override = None ) → DeviceMesh [source] Initializes a DeviceMesh based on device_type and mesh_shape (torch parity). This creates a DeviceMesh with an n-dimensional array layout, where n is the length of mesh_shape . If mesh_dim_names is provided, each dimension is labeled as mesh_dim_names[i] . Note Follows SPMD: ensure mesh_shape is identical across all ranks. Example: >>> mesh_1d = init_device_mesh ( "cuda" , mesh_shape = ( 8 ,)) >>> mesh_2d = init_device_mesh ( "cuda" , mesh_shape = ( 2 , 8 ), ... mesh_dim_names = ( "dp" , "tp" )) # tensorplay.distributed.distributed_c10d.all_gather_object URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.all_gather_object.html tensorplay.distributed.distributed_c10d.all_gather_object tensorplay.distributed.distributed_c10d. all_gather_object ( object_list , obj , group = None ) → None [source] Gathers picklable objects from the whole group into a list (torch parity). # tensorplay.distributed.distributed_c10d.all_gather URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.all_gather.html tensorplay.distributed.distributed_c10d.all_gather tensorplay.distributed.distributed_c10d. all_gather ( tensor_list : List [ TensorBase ] , tensor : TensorBase , group = None , async_op : bool = False ) [source] # tensorplay.distributed.distributed_c10d.all_reduce URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.all_reduce.html tensorplay.distributed.distributed_c10d.all_reduce tensorplay.distributed.distributed_c10d. all_reduce ( tensor : TensorBase , op : int = 0 , group = None , async_op : bool = False ) [source] # tensorplay.distributed.distributed_c10d.all_to_all_single URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.all_to_all_single.html tensorplay.distributed.distributed_c10d.all_to_all_single tensorplay.distributed.distributed_c10d. all_to_all_single ( output : TensorBase , input : TensorBase , output_split_sizes : List [ int ] | None = None , input_split_sizes : List [ int ] | None = None , group = None , async_op : bool = False ) [source] Splits input evenly (or by split sizes) and scatters the chunks. Split sizes are in group-rank order, matching torch semantics. # tensorplay.distributed.distributed_c10d.all_to_all URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.all_to_all.html tensorplay.distributed.distributed_c10d.all_to_all tensorplay.distributed.distributed_c10d. all_to_all ( output_tensor_list : List [ TensorBase ] , input_tensor_list : List [ TensorBase ] , group = None , async_op : bool = False ) [source] Scatters a list of tensors to ranks and collects one from each. Mirrors ProcessGroupNCCL::alltoall: per-rank splits are the tensor numels, executed as one grouped send/recv exchange. # Backend URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.Backend.html Backend class tensorplay.distributed.distributed_c10d. Backend [source] # tensorplay.distributed.distributed_c10d.barrier URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.barrier.html tensorplay.distributed.distributed_c10d.barrier tensorplay.distributed.distributed_c10d. barrier ( group = None , async_op : bool = False , device_ids : List [ int ] | None = None ) [source] # tensorplay.distributed.distributed_c10d.batch_isend_irecv URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.batch_isend_irecv.html tensorplay.distributed.distributed_c10d.batch_isend_irecv tensorplay.distributed.distributed_c10d. batch_isend_irecv ( p2p_op_list : List [ P2POp ] ) → List [ Work ] [source] Send or receive a batch of tensors asynchronously (torch parity). All operations are treated as a single NCCL group so ordering of sends vs receives cannot deadlock. Every rank in group must participate. # tensorplay.distributed.distributed_c10d.broadcast_object_list URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.broadcast_object_list.html tensorplay.distributed.distributed_c10d.broadcast_object_list tensorplay.distributed.distributed_c10d. broadcast_object_list ( object_list : list , src : int = 0 , group = None , device = None ) → None [source] Broadcasts picklable objects in object_list to the whole group. src is a global rank (torch semantics). # tensorplay.distributed.distributed_c10d.broadcast URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.broadcast.html tensorplay.distributed.distributed_c10d.broadcast tensorplay.distributed.distributed_c10d. broadcast ( tensor : TensorBase , src : int , group = None , async_op : bool = False ) [source] # tensorplay.distributed.distributed_c10d.destroy_process_group URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.destroy_process_group.html tensorplay.distributed.distributed_c10d.destroy_process_group tensorplay.distributed.distributed_c10d. destroy_process_group ( group = None ) → None [source] # tensorplay.distributed.distributed_c10d.gather_object URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.gather_object.html tensorplay.distributed.distributed_c10d.gather_object tensorplay.distributed.distributed_c10d. gather_object ( obj , object_gather_list = None , dst : int | None = None , group = None ) → None [source] Gathers picklable objects from the whole group in a single process. dst is a global rank (torch semantics). # tensorplay.distributed.distributed_c10d.gather URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.gather.html tensorplay.distributed.distributed_c10d.gather tensorplay.distributed.distributed_c10d. gather ( tensor : TensorBase , gather_list : List [ TensorBase ] | None = None , dst : int = 0 , group = None , async_op : bool = False ) [source] # tensorplay.distributed.distributed_c10d.get_backend URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.get_backend.html tensorplay.distributed.distributed_c10d.get_backend tensorplay.distributed.distributed_c10d. get_backend ( group = None ) → str [source] # tensorplay.distributed.distributed_c10d.get_global_rank URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.get_global_rank.html tensorplay.distributed.distributed_c10d.get_global_rank tensorplay.distributed.distributed_c10d. get_global_rank ( group : ProcessGroup , group_rank : int ) → int [source] Translate a group rank into a global rank (torch parity). # tensorplay.distributed.distributed_c10d.get_group_rank URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.get_group_rank.html tensorplay.distributed.distributed_c10d.get_group_rank tensorplay.distributed.distributed_c10d. get_group_rank ( group : ProcessGroup , global_rank : int ) → int [source] Translate a global rank into a group rank (torch parity). # tensorplay.distributed.distributed_c10d.get_process_group_ranks URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.get_process_group_ranks.html tensorplay.distributed.distributed_c10d.get_process_group_ranks tensorplay.distributed.distributed_c10d. get_process_group_ranks ( group ) → List [ int ] [source] Get all ranks associated with group (torch parity). # tensorplay.distributed.distributed_c10d.get_rank URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.get_rank.html tensorplay.distributed.distributed_c10d.get_rank tensorplay.distributed.distributed_c10d. get_rank ( group = None ) → int [source] # tensorplay.distributed.distributed_c10d.get_world_size URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.get_world_size.html tensorplay.distributed.distributed_c10d.get_world_size tensorplay.distributed.distributed_c10d. get_world_size ( group = None ) → int [source] # tensorplay.distributed.distributed_c10d.init_process_group URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.init_process_group.html tensorplay.distributed.distributed_c10d.init_process_group tensorplay.distributed.distributed_c10d. init_process_group ( backend : str | None = None , init_method : str | None = None , world_size : int = -1 , rank : int = -1 , store = None , group_name : str = '' , timeout : timedelta = datetime.timedelta(seconds=1800) ) → None [source] # tensorplay.distributed.distributed_c10d.irecv URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.irecv.html tensorplay.distributed.distributed_c10d.irecv tensorplay.distributed.distributed_c10d. irecv ( tensor : TensorBase , src : int | None = None , group = None , tag : int = 0 ) [source] Receives a tensor asynchronously (torch parity; src global rank). Returns a Work handle whose wait() completes the copy, or None if not part of the group. # tensorplay.distributed.distributed_c10d.is_available URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.is_available.html tensorplay.distributed.distributed_c10d.is_available tensorplay.distributed.distributed_c10d. is_available ( ) → bool [source] # tensorplay.distributed.distributed_c10d.is_gloo_available URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.is_gloo_available.html tensorplay.distributed.distributed_c10d.is_gloo_available tensorplay.distributed.distributed_c10d. is_gloo_available ( ) → bool [source] # tensorplay.distributed.distributed_c10d.is_initialized URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.is_initialized.html tensorplay.distributed.distributed_c10d.is_initialized tensorplay.distributed.distributed_c10d. is_initialized ( ) → bool [source] # tensorplay.distributed.distributed_c10d.is_mpi_available URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.is_mpi_available.html tensorplay.distributed.distributed_c10d.is_mpi_available tensorplay.distributed.distributed_c10d. is_mpi_available ( ) → bool [source] # tensorplay.distributed.distributed_c10d.is_nccl_available URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.is_nccl_available.html tensorplay.distributed.distributed_c10d.is_nccl_available tensorplay.distributed.distributed_c10d. is_nccl_available ( ) → bool [source] # tensorplay.distributed.distributed_c10d.isend URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.isend.html tensorplay.distributed.distributed_c10d.isend tensorplay.distributed.distributed_c10d. isend ( tensor : TensorBase , dst : int , group = None , tag : int = 0 ) [source] Send a tensor asynchronously (torch parity; dst is a global rank). Returns a Work handle, or None if not part of the group. # tensorplay.distributed.distributed_c10d.new_group URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.new_group.html tensorplay.distributed.distributed_c10d.new_group tensorplay.distributed.distributed_c10d. new_group ( ranks : List [ int ] | None = None , timeout : timedelta = datetime.timedelta(seconds=1800) , backend : str | None = None , pg_options = None ) [source] # P2POp URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.P2POp.html P2POp class tensorplay.distributed.distributed_c10d. P2POp ( op , tensor , peer : int | None = None , group = None , tag : int = 0 , group_peer : int | None = None ) [source] A class to build point-to-point operations for batch_isend_irecv . Args mirror torch: op is isend() or irecv() , peer is a global rank (or group_peer a group rank). # tensorplay.distributed.distributed_c10d.recv_object_list URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.recv_object_list.html tensorplay.distributed.distributed_c10d.recv_object_list tensorplay.distributed.distributed_c10d. recv_object_list ( object_list : list , src : int | None = None , group = None , device = None ) → int [source] Receives picklable objects in object_list synchronously (torch parity). Returns the sender’s global rank. # tensorplay.distributed.distributed_c10d.recv URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.recv.html tensorplay.distributed.distributed_c10d.recv tensorplay.distributed.distributed_c10d. recv ( tensor : TensorBase , src : int | None = None , group = None , tag : int = 0 ) [source] Receives a tensor synchronously; returns the sender rank. # tensorplay.distributed.distributed_c10d.reduce_scatter URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.reduce_scatter.html tensorplay.distributed.distributed_c10d.reduce_scatter tensorplay.distributed.distributed_c10d. reduce_scatter ( output : TensorBase , input_list : List [ TensorBase ] , op : int = 0 , group = None , async_op : bool = False ) [source] # tensorplay.distributed.distributed_c10d.reduce URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.reduce.html tensorplay.distributed.distributed_c10d.reduce tensorplay.distributed.distributed_c10d. reduce ( tensor : TensorBase , dst : int , op : int = 0 , group = None , async_op : bool = False ) [source] # ReduceOp URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.ReduceOp.html ReduceOp class tensorplay.distributed.distributed_c10d. ReduceOp [source] # tensorplay.distributed.distributed_c10d.scatter_object_list URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.scatter_object_list.html tensorplay.distributed.distributed_c10d.scatter_object_list tensorplay.distributed.distributed_c10d. scatter_object_list ( scatter_object_output_list : list , scatter_object_input_list : Sequence [ object ] | None = None , src : int = 0 , group = None ) → None [source] Scatters picklable objects to the whole group (torch parity). src is a global rank. On each rank the scattered object is stored as the first element of scatter_object_output_list . # tensorplay.distributed.distributed_c10d.scatter URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.scatter.html tensorplay.distributed.distributed_c10d.scatter tensorplay.distributed.distributed_c10d. scatter ( tensor : TensorBase , scatter_list : List [ TensorBase ] | None = None , src : int = 0 , group = None , async_op : bool = False ) [source] # tensorplay.distributed.distributed_c10d.send_object_list URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.send_object_list.html tensorplay.distributed.distributed_c10d.send_object_list tensorplay.distributed.distributed_c10d. send_object_list ( object_list : Sequence [ object ] , dst : int , group = None , device = None ) → None [source] Sends picklable objects in object_list synchronously (torch parity). # tensorplay.distributed.distributed_c10d.send URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.send.html tensorplay.distributed.distributed_c10d.send tensorplay.distributed.distributed_c10d. send ( tensor : TensorBase , dst : int , group = None , tag : int = 0 ) [source] Send a tensor synchronously (torch parity). # Work URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.distributed_c10d.Work.html Work class tensorplay.distributed.distributed_c10d. Work ( event , done = None , tensors = None ) [source] Handle for an async collective (torch.distributed.Work subset). get_future() mirrors c10d: it returns a tensorplay.futures.Future resolving to the list of output tensors once the collective completes. # tensorplay.distributed.launch.launch URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.launch.launch.html tensorplay.distributed.launch.launch tensorplay.distributed.launch. launch ( args ) [source] # tensorplay.distributed.launch.main URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.launch.main.html tensorplay.distributed.launch.main tensorplay.distributed.launch. main ( args = None ) [source] # tensorplay.distributed.launch.parse_args URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.launch.parse_args.html tensorplay.distributed.launch.parse_args tensorplay.distributed.launch. parse_args ( args ) [source] # tensorplay.distributed.rendezvous.register_rendezvous_handler URL: https://www.tensorplay.cn/docs/generated/tensorplay.distributed.rendezvous.register_rendezvous_handler.html tensorplay.distributed.rendezvous.register_rendezvous_handler tensorplay.distributed.rendezvous. register_rendezvous_handler ( scheme , handler ) [source] Register a new rendezvous handler. Before we can run collective algorithms, participating processes need to find each other and exchange information to be able to communicate. We call this process rendezvous. The outcome of the rendezvous process is a triplet containing a shared key/value store, the rank of the process, and the total number of participating processes. If none of the bundled rendezvous methods apply to your execution environment you can opt to register your own rendezvous handler. Pick a unique name and use the URL scheme to identify it when calling the rendezvous() function. Parameters : scheme ( str ) – URL scheme to identify your rendezvous handler. handler ( function ) – Handler that is invoked when the rendezvous() function is called with a URL that uses the corresponding scheme. It must be a generator function that yields the triplet. # tensorplay.div_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.div_.html tensorplay.div_ tensorplay. div_ ( input , other ) [source] # dtype URL: https://www.tensorplay.cn/docs/generated/tensorplay.dtype.html dtype tensorplay. dtype alias of DType # DType URL: https://www.tensorplay.cn/docs/generated/tensorplay.DType.html DType class tensorplay. DType Members: uint8 int8 int16 int32 int64 uint16 uint32 uint64 float16 bfloat16 float32 float64 complex32 complex64 complex128 bcomplex32 bool undefined property name # tensorplay.elu URL: https://www.tensorplay.cn/docs/generated/tensorplay.elu.html tensorplay.elu tensorplay. elu ( input , alpha = 1 , scale = 1 , input_scale = 1 ) [source] # tensorplay.expand_as URL: https://www.tensorplay.cn/docs/generated/tensorplay.expand_as.html tensorplay.expand_as tensorplay. expand_as ( input , other ) [source] # tensorplay.expand URL: https://www.tensorplay.cn/docs/generated/tensorplay.expand.html tensorplay.expand tensorplay. expand ( input , size , implicit = False ) [source] # tensorplay.exponential_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.exponential_.html tensorplay.exponential_ tensorplay. exponential_ ( input , lambd = 1.0 ) [source] # tensorplay.fft_fft URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft_fft.html tensorplay.fft_fft tensorplay. fft_fft ( input , n = -1 , dim = -1 , norm = 'backward' ) [source] # tensorplay.fft_ifft URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft_ifft.html tensorplay.fft_ifft tensorplay. fft_ifft ( input , n = -1 , dim = -1 , norm = 'backward' ) [source] # tensorplay.fft_irfft URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft_irfft.html tensorplay.fft_irfft tensorplay. fft_irfft ( input , n = -1 , dim = -1 , norm = 'backward' ) [source] # tensorplay.fft_rfft URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft_rfft.html tensorplay.fft_rfft tensorplay. fft_rfft ( input , n = -1 , dim = -1 , norm = 'backward' ) [source] # tensorplay.fft.fft URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft.fft.html tensorplay.fft.fft tensorplay.fft. fft ( input , n = None , dim = -1 , norm = None ) [source] Computes the one-dimensional discrete Fourier transform. Parameters : input ( Tensor ) – the input tensor n ( int , optional ) – signal length; zero-pads/truncates dim dim ( int , optional ) – the dimension to transform. Default: -1 norm ( str , optional ) – "backward" , "forward" or "ortho" . Default: None (= "backward" ) # tensorplay.fft.fft2 URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft.fft2.html tensorplay.fft.fft2 tensorplay.fft. fft2 ( input , s = None , dim = (-2, -1) , norm = None ) [source] Computes the two-dimensional discrete Fourier transform. Equivalent to stacked 1-D fft() calls along each transformed dim. # tensorplay.fft.fftfreq URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft.fftfreq.html tensorplay.fft.fftfreq tensorplay.fft. fftfreq ( n , d=1.0 , * , dtype= , device=None ) [source] DFT sample frequencies (cycles/unit): [0, 1, ..., n/2-1, -n/2, ..., -1] / (n*d) . Parameters : n ( int ) – window length d ( float , optional ) – sample spacing. Default: 1.0 device ( dtype / ) – forwarded to the factory ops. Default: float32/CPU # tensorplay.fft.fftn URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft.fftn.html tensorplay.fft.fftn tensorplay.fft. fftn ( input , s = None , dim = None , norm = None ) [source] Computes the N-dimensional discrete Fourier transform over dim . # tensorplay.fft.fftshift URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft.fftshift.html tensorplay.fft.fftshift tensorplay.fft. fftshift ( input , dim = None ) [source] Re-orders an N-D FFT output so the zero-frequency term is centered. Shifts by +n // 2 along each (or the given) dimension(s). # tensorplay.fft.hfft URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft.hfft.html tensorplay.fft.hfft tensorplay.fft. hfft ( input , n = None , dim = -1 , norm = None ) [source] Computes the 1-D FFT of a Hermitian-symmetric spectrum; real output. Equivalent to irfft() applied to input.conj() ; n is the output length (default 2 * (input.size(dim) - 1) ). # tensorplay.fft.hfft2 URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft.hfft2.html tensorplay.fft.hfft2 tensorplay.fft. hfft2 ( input , s = None , dim = (-2, -1) , norm = None ) [source] Two-dimensional inverse of a Hermitian-symmetric spectrum; real output. # tensorplay.fft.hfftn URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft.hfftn.html tensorplay.fft.hfftn tensorplay.fft. hfftn ( input , s = None , dim = None , norm = None ) [source] N-dimensional FFT of a Hermitian-symmetric spectrum; real output. Applies hfft() (conjugate + complex-to-real) along the final transformed dimension, then ifft() over the remaining dimensions. # tensorplay.fft.ifft URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft.ifft.html tensorplay.fft.ifft tensorplay.fft. ifft ( input , n = None , dim = -1 , norm = None ) [source] Computes the one-dimensional inverse discrete Fourier transform. # tensorplay.fft.ifft2 URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft.ifft2.html tensorplay.fft.ifft2 tensorplay.fft. ifft2 ( input , s = None , dim = (-2, -1) , norm = None ) [source] Computes the two-dimensional inverse discrete Fourier transform. # tensorplay.fft.ifftn URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft.ifftn.html tensorplay.fft.ifftn tensorplay.fft. ifftn ( input , s = None , dim = None , norm = None ) [source] Computes the N-dimensional inverse discrete Fourier transform. # tensorplay.fft.ifftshift URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft.ifftshift.html tensorplay.fft.ifftshift tensorplay.fft. ifftshift ( input , dim = None ) [source] Inverse of fftshift() ; shifts by -(n // 2) (odd-safe). # tensorplay.fft.ihfft URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft.ihfft.html tensorplay.fft.ihfft tensorplay.fft. ihfft ( input , n = None , dim = -1 , norm = None ) [source] Computes the inverse of hfft() ; one-sided complex output. Equivalent to rfft() applied to input.conj() ; n zero-pads/truncates the real input along dim . # tensorplay.fft.ihfft2 URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft.ihfft2.html tensorplay.fft.ihfft2 tensorplay.fft. ihfft2 ( input , s = None , dim = (-2, -1) , norm = None ) [source] Two-dimensional counterpart of ihfft() . # tensorplay.fft.ihfftn URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft.ihfftn.html tensorplay.fft.ihfftn tensorplay.fft. ihfftn ( input , s = None , dim = None , norm = None ) [source] Inverse of hfftn() : ihfft() along the final transformed dimension, then fft() over the remaining dimensions. # tensorplay.fft.irfft URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft.irfft.html tensorplay.fft.irfft tensorplay.fft. irfft ( input , n = None , dim = -1 , norm = None ) [source] Computes the inverse of rfft() ; n is the output length. # tensorplay.fft.irfft2 URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft.irfft2.html tensorplay.fft.irfft2 tensorplay.fft. irfft2 ( input , s = None , dim = (-2, -1) , norm = None ) [source] Inverse of rfft2() : irfft() on the final dimension first ( s[-1] is the real output size), then ifft() on the rest. # tensorplay.fft.irfftn URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft.irfftn.html tensorplay.fft.irfftn tensorplay.fft. irfftn ( input , s = None , dim = None , norm = None , * , out = None ) [source] Inverse of rfftn() ; s[-1] is the real output size. # tensorplay.fft.rfft URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft.rfft.html tensorplay.fft.rfft tensorplay.fft. rfft ( input , n = None , dim = -1 , norm = None ) [source] Computes the one-dimensional FFT of real input, one-sided output. # tensorplay.fft.rfft2 URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft.rfft2.html tensorplay.fft.rfft2 tensorplay.fft. rfft2 ( input , s = None , dim = (-2, -1) , norm = None ) [source] Two-dimensional FFT of real input: fft() on leading dims, then rfft() on the final transformed dimension. # tensorplay.fft.rfftfreq URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft.rfftfreq.html tensorplay.fft.rfftfreq tensorplay.fft. rfftfreq ( n , d=1.0 , * , dtype= , device=None ) [source] Sample frequencies for rfft() /one-sided transforms: [0..n//2] / (n*d) . # tensorplay.fft.rfftn URL: https://www.tensorplay.cn/docs/generated/tensorplay.fft.rfftn.html tensorplay.fft.rfftn tensorplay.fft. rfftn ( input , s = None , dim = None , norm = None , * , out = None ) [source] N-dimensional FFT of real input; one-sided along the last listed dim. # tensorplay.fork_rng URL: https://www.tensorplay.cn/docs/generated/tensorplay.fork_rng.html tensorplay.fork_rng tensorplay. fork_rng ( devices = None , enabled = True , _caller = 'fork_rng' , _devices_kw = 'devices' ) [source] Forks the RNG state: code inside the context gets a pristine RNG. Saves the CPU RNG state on entry and restores it on exit, so random operations inside the block do not advance the outer stream (mirrors torch.random.fork_rng ). CUDA devices are not forked; pass devices=[] explicitly to acknowledge that (same contract as torch). # tensorplay.forward_add URL: https://www.tensorplay.cn/docs/generated/tensorplay.forward_add.html tensorplay.forward_add tensorplay. forward_add ( a , da , b , db ) [source] # tensorplay.forward_cos URL: https://www.tensorplay.cn/docs/generated/tensorplay.forward_cos.html tensorplay.forward_cos tensorplay. forward_cos ( a , da ) [source] # tensorplay.forward_div URL: https://www.tensorplay.cn/docs/generated/tensorplay.forward_div.html tensorplay.forward_div tensorplay. forward_div ( a , da , b , db ) [source] # tensorplay.forward_exp URL: https://www.tensorplay.cn/docs/generated/tensorplay.forward_exp.html tensorplay.forward_exp tensorplay. forward_exp ( a , da ) [source] # tensorplay.forward_log URL: https://www.tensorplay.cn/docs/generated/tensorplay.forward_log.html tensorplay.forward_log tensorplay. forward_log ( a , da ) [source] # tensorplay.forward_mm URL: https://www.tensorplay.cn/docs/generated/tensorplay.forward_mm.html tensorplay.forward_mm tensorplay. forward_mm ( a , da , b , db ) [source] # tensorplay.forward_mul URL: https://www.tensorplay.cn/docs/generated/tensorplay.forward_mul.html tensorplay.forward_mul tensorplay. forward_mul ( a , da , b , db ) [source] # tensorplay.forward_neg URL: https://www.tensorplay.cn/docs/generated/tensorplay.forward_neg.html tensorplay.forward_neg tensorplay. forward_neg ( a , da ) [source] # tensorplay.forward_pow URL: https://www.tensorplay.cn/docs/generated/tensorplay.forward_pow.html tensorplay.forward_pow tensorplay. forward_pow ( a , da , b , db ) [source] # tensorplay.forward_relu URL: https://www.tensorplay.cn/docs/generated/tensorplay.forward_relu.html tensorplay.forward_relu tensorplay. forward_relu ( a , da ) [source] # tensorplay.forward_sigmoid URL: https://www.tensorplay.cn/docs/generated/tensorplay.forward_sigmoid.html tensorplay.forward_sigmoid tensorplay. forward_sigmoid ( a , da ) [source] # tensorplay.forward_sin URL: https://www.tensorplay.cn/docs/generated/tensorplay.forward_sin.html tensorplay.forward_sin tensorplay. forward_sin ( a , da ) [source] # tensorplay.forward_sqrt URL: https://www.tensorplay.cn/docs/generated/tensorplay.forward_sqrt.html tensorplay.forward_sqrt tensorplay. forward_sqrt ( a , da ) [source] # tensorplay.forward_sub URL: https://www.tensorplay.cn/docs/generated/tensorplay.forward_sub.html tensorplay.forward_sub tensorplay. forward_sub ( a , da , b , db ) [source] # tensorplay.forward_tanh URL: https://www.tensorplay.cn/docs/generated/tensorplay.forward_tanh.html tensorplay.forward_tanh tensorplay. forward_tanh ( a , da ) [source] # tensorplay.from_dlpack URL: https://www.tensorplay.cn/docs/generated/tensorplay.from_dlpack.html tensorplay.from_dlpack tensorplay. from_dlpack ( obj : object ) → tensorplay::Tensor # tensorplay.functional.abs_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.abs_.html tensorplay.functional.abs_ tensorplay.functional. abs_ ( input ) [source] # tensorplay.functional.abs URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.abs.html tensorplay.functional.abs tensorplay.functional. abs ( input ) [source] # tensorplay.functional.acos URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.acos.html tensorplay.functional.acos tensorplay.functional. acos ( input ) [source] # tensorplay.functional.acosh URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.acosh.html tensorplay.functional.acosh tensorplay.functional. acosh ( input ) [source] # tensorplay.functional.add URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.add.html tensorplay.functional.add tensorplay.functional. add ( input , other , alpha = 1 ) [source] # tensorplay.functional.addbmm URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.addbmm.html tensorplay.functional.addbmm tensorplay.functional. addbmm ( input , batch1 , batch2 , beta = 1 , alpha = 1 ) [source] # tensorplay.functional.addcdiv URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.addcdiv.html tensorplay.functional.addcdiv tensorplay.functional. addcdiv ( input , tensor1 , tensor2 , value = 1 ) [source] # tensorplay.functional.addcmul URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.addcmul.html tensorplay.functional.addcmul tensorplay.functional. addcmul ( input , tensor1 , tensor2 , value = 1 ) [source] # tensorplay.functional.addmm URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.addmm.html tensorplay.functional.addmm tensorplay.functional. addmm ( input , mat1 , mat2 , beta = 1 , alpha = 1 ) [source] # tensorplay.functional.addmv URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.addmv.html tensorplay.functional.addmv tensorplay.functional. addmv ( input , mat , vec , beta = 1 , alpha = 1 ) [source] # tensorplay.functional.addr URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.addr.html tensorplay.functional.addr tensorplay.functional. addr ( input , vec1 , vec2 , beta = 1 , alpha = 1 ) [source] # tensorplay.functional.airy_ai URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.airy_ai.html tensorplay.functional.airy_ai tensorplay.functional. airy_ai ( input ) [source] # tensorplay.functional.all URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.all.html tensorplay.functional.all tensorplay.functional. all ( input ) [source] # tensorplay.functional.allclose URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.allclose.html tensorplay.functional.allclose tensorplay.functional. allclose ( input , other , rtol = 1e-05 , atol = 1e-08 , equal_nan = False ) [source] # tensorplay.functional.amax URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.amax.html tensorplay.functional.amax tensorplay.functional. amax ( input , dim = [] , keepdim = False ) [source] # tensorplay.functional.amin URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.amin.html tensorplay.functional.amin tensorplay.functional. amin ( input , dim = [] , keepdim = False ) [source] # tensorplay.functional.aminmax URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.aminmax.html tensorplay.functional.aminmax tensorplay.functional. aminmax ( input , dim = [] , keepdim = False ) [source] # tensorplay.functional.angle URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.angle.html tensorplay.functional.angle tensorplay.functional. angle ( input ) [source] # tensorplay.functional.any URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.any.html tensorplay.functional.any tensorplay.functional. any ( input ) [source] # tensorplay.functional.arange URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.arange.html tensorplay.functional.arange tensorplay.functional. arange ( *args , dtype= , device=None , requires_grad=False ) [source] # tensorplay.functional.argmax URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.argmax.html tensorplay.functional.argmax tensorplay.functional. argmax ( input , dim = None , keepdim = False ) [source] # tensorplay.functional.argmin URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.argmin.html tensorplay.functional.argmin tensorplay.functional. argmin ( input , dim = None , keepdim = False ) [source] # tensorplay.functional.argsort URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.argsort.html tensorplay.functional.argsort tensorplay.functional. argsort ( input , dim = -1 , descending = False ) [source] # tensorplay.functional.argwhere URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.argwhere.html tensorplay.functional.argwhere tensorplay.functional. argwhere ( input ) [source] # tensorplay.functional.asin URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.asin.html tensorplay.functional.asin tensorplay.functional. asin ( input ) [source] # tensorplay.functional.asinh URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.asinh.html tensorplay.functional.asinh tensorplay.functional. asinh ( input ) [source] # tensorplay.functional.atan URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.atan.html tensorplay.functional.atan tensorplay.functional. atan ( input ) [source] # tensorplay.functional.atan2 URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.atan2.html tensorplay.functional.atan2 tensorplay.functional. atan2 ( input , other ) [source] # tensorplay.functional.atanh URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.atanh.html tensorplay.functional.atanh tensorplay.functional. atanh ( input ) [source] # tensorplay.functional.gcd URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.gcd.html tensorplay.functional.gcd tensorplay.functional. gcd ( input , other ) [source] # tensorplay.functional.isfinite URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.isfinite.html tensorplay.functional.isfinite tensorplay.functional. isfinite ( input ) [source] # tensorplay.functional.logspace URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.logspace.html tensorplay.functional.logspace tensorplay.functional. logspace ( start , end , steps , base=10.0 , dtype= , device=None , requires_grad=False ) [source] # tensorplay.functional.logsumexp URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.logsumexp.html tensorplay.functional.logsumexp tensorplay.functional. logsumexp ( input , dim , keepdim = False ) [source] # tensorplay.functional.lstm URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.lstm.html tensorplay.functional.lstm tensorplay.functional. lstm ( input , hx , params , has_biases = True , num_layers = 1 , dropout_p = 0.0 , training = False , bidirectional = False , batch_first = False ) [source] # tensorplay.functional.narrow URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.narrow.html tensorplay.functional.narrow tensorplay.functional. narrow ( input , dim , start , length ) [source] # tensorplay.functional.ne URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.ne.html tensorplay.functional.ne tensorplay.functional. ne ( input , other ) [source] # tensorplay.functional.neg_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.neg_.html tensorplay.functional.neg_ tensorplay.functional. neg_ ( input ) [source] # tensorplay.functional.neg URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.neg.html tensorplay.functional.neg tensorplay.functional. neg ( input ) [source] # tensorplay.functional.randint_like URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.randint_like.html tensorplay.functional.randint_like tensorplay.functional. randint_like ( input , low , high , dtype= , device=None , requires_grad=False ) [source] # tensorplay.functional.randint URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.randint.html tensorplay.functional.randint tensorplay.functional. randint ( low , high , size , dtype= , device=None , requires_grad=False ) [source] # tensorplay.functional.randn_like URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.randn_like.html tensorplay.functional.randn_like tensorplay.functional. randn_like ( input , dtype= , device=None , requires_grad=False ) [source] # tensorplay.functional.randn URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.randn.html tensorplay.functional.randn tensorplay.functional. randn ( * size , dtype = None , device = None , requires_grad = False ) [source] # tensorplay.functional.randperm URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.randperm.html tensorplay.functional.randperm tensorplay.functional. randperm ( n , dtype= , device=None , requires_grad=False ) [source] # tensorplay.functional.shifted_chebyshev_polynomial_v URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.shifted_chebyshev_polynomial_v.html tensorplay.functional.shifted_chebyshev_polynomial_v tensorplay.functional. shifted_chebyshev_polynomial_v ( x , n ) [source] # tensorplay.functional.shifted_chebyshev_polynomial_w URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.shifted_chebyshev_polynomial_w.html tensorplay.functional.shifted_chebyshev_polynomial_w tensorplay.functional. shifted_chebyshev_polynomial_w ( x , n ) [source] # tensorplay.functional.sigmoid URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.sigmoid.html tensorplay.functional.sigmoid tensorplay.functional. sigmoid ( input ) [source] # tensorplay.functional.sign URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.sign.html tensorplay.functional.sign tensorplay.functional. sign ( input ) [source] # tensorplay.functional.signbit URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.signbit.html tensorplay.functional.signbit tensorplay.functional. signbit ( input ) [source] # tensorplay.functional.tan URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.tan.html tensorplay.functional.tan tensorplay.functional. tan ( input ) [source] # tensorplay.functional.tanh URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.tanh.html tensorplay.functional.tanh tensorplay.functional. tanh ( input ) [source] # tensorplay.functional.tensor_split URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.tensor_split.html tensorplay.functional.tensor_split tensorplay.functional. tensor_split ( input , sections , dim = 0 ) [source] # tensorplay.functional.threshold URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.threshold.html tensorplay.functional.threshold tensorplay.functional. threshold ( input , threshold , value ) [source] # tensorplay.functional.tile URL: https://www.tensorplay.cn/docs/generated/tensorplay.functional.tile.html tensorplay.functional.tile tensorplay.functional. tile ( input , dims ) [source] # Future URL: https://www.tensorplay.cn/docs/generated/tensorplay.futures.Future.html Future class tensorplay.futures. Future ( * , _completer : Callable [ [ ] , Any ] | None = None ) [source] Holder for an asynchronous result (torch parity subset). Example: >>> fut = tp . futures . Future () >>> fut . set_result ( 1 ) >>> fut . value () 1 >>> chained = other . then ( lambda f : f . value ()[ 0 ]) >>> chained . value () tensor(...) add_done_callback ( callback : Callable [ [ Future ] , Any ] ) → None [source] Appends a callback run when this future completes. done ( ) → bool [source] Returns whether the future is complete (non-blocking). is_done ( ) → bool [source] Alias of done() . set_result ( result : Any ) → None [source] Sets the result value and fires callbacks (torch parity). then ( callback : Callable [ [ Future ] , Any ] ) → Future [source] Adds a callback mapped over this future; returns the new future. The callback receives this future and its return value becomes the derived future’s result (torch Future.then contract). value ( ) → Any [source] Gets the result value, blocking until it completes. Raises RuntimeError on error results (mirrors torch’s ValueError -on-error behavior loosely; tp stores exceptions as results and re-raises them here). wait ( ) → Any [source] Blocks until complete and returns the result value. # tensorplay.gammainc URL: https://www.tensorplay.cn/docs/generated/tensorplay.gammainc.html tensorplay.gammainc tensorplay. gammainc ( a , x ) [source] # tensorplay.gammaincc URL: https://www.tensorplay.cn/docs/generated/tensorplay.gammaincc.html tensorplay.gammaincc tensorplay. gammaincc ( a , x ) [source] # tensorplay.gelu URL: https://www.tensorplay.cn/docs/generated/tensorplay.gelu.html tensorplay.gelu tensorplay. gelu ( input , approximate = 'none' ) [source] # Generator URL: https://www.tensorplay.cn/docs/generated/tensorplay.Generator.html Generator class tensorplay. Generator get_state ( self : tensorplay._C.Generator ) → tensorplay::Tensor initial_seed ( self : tensorplay._C.Generator ) → int manual_seed ( self : tensorplay._C.Generator , seed : SupportsInt | SupportsIndex ) → int seed ( self : tensorplay._C.Generator ) → int set_state ( self: tensorplay._C.Generator , new_state: tensorplay::Tensor ) → None # tensorplay.geometric_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.geometric_.html tensorplay.geometric_ tensorplay. geometric_ ( input , p ) [source] # GradScaler URL: https://www.tensorplay.cn/docs/generated/tensorplay.GradScaler.html GradScaler class tensorplay. GradScaler ( device : str = 'cuda' , init_scale : float = 65536.0 , growth_factor : float = 2.0 , backoff_factor : float = 0.5 , growth_interval : int = 2000 , enabled : bool = True ) [source] An instance scaler of GradScaler . Helps perform the steps of gradient scaling conveniently. scaler.scale(loss) multiplies a given loss by scaler ’s current scale factor. scaler.step(optimizer) safely unscales gradients and calls optimizer.step() . scaler.update() updates scaler ’s scale factor. Example: # Creates a GradScaler once at the beginning of training. scaler = GradScaler () for epoch in epochs : for input , target in data : optimizer . zero_grad () output = model ( input ) loss = loss_fn ( output , target ) # Scales loss. Calls backward() on scaled loss to create scaled gradients. scaler . scale ( loss ) . backward () # scaler.step() first unscales gradients of the optimizer's params. # If gradients don't contain infs/NaNs, optimizer.step() is then called, # otherwise, optimizer.step() is skipped. scaler . step ( optimizer ) # Updates the scale for next iteration. scaler . update () scaler dynamically estimates the scale factor each iteration. To minimize gradient underflow, a large scale factor should be used. However, float16 values can “overflow” (become inf or NaN) if the scale factor is too large. Therefore, the optimal scale factor is the largest factor that can be used without incurring inf or NaN gradient values. scaler approximates the optimal scale factor over time by checking the gradients for infs and NaNs during every scaler.step(optimizer) (or optional separate scaler.unscale_(optimizer) , see unscale_() ). If infs/NaNs are found, scaler.step(optimizer) skips the underlying optimizer.step() (so the params themselves remain uncorrupted) and update() multiplies the scale by backoff_factor . If no infs/NaNs are found, scaler.step(optimizer) runs the underlying optimizer.step() as usual. If growth_interval unskipped iterations occur consecutively, update() multiplies the scale by growth_factor . Parameters : device ( str , optional , default="cuda" ) – Device type to use. Possible values are: ‘cuda’ and ‘cpu’. The type is the same as the type attribute of a tensorplay.device . Thus, you may obtain the device type of a tensor using Tensor.device.type . init_scale ( float , optional , default=2.**16 ) – Initial scale factor. growth_factor ( float , optional , default=2.0 ) – Factor by which the scale is multiplied during update() if no inf/NaN gradients occur for growth_interval consecutive iterations. backoff_factor ( float , optional , default=0.5 ) – Factor by which the scale is multiplied during update() if inf/NaN gradients occur in an iteration. growth_interval ( int , optional , default=2000 ) – Number of consecutive iterations without inf/NaN gradients that must occur for the scale to be multiplied by growth_factor . enabled ( bool , optional ) – If False , disables gradient scaling. step() simply invokes the underlying optimizer.step() , and other methods become no-ops. Default: True get_backoff_factor ( ) → float [source] Return a Python float containing the scale backoff factor. get_growth_factor ( ) → float [source] Return a Python float containing the scale growth factor. get_growth_interval ( ) → int [source] Return a Python int containing the growth interval. get_scale ( ) → float [source] Return a Python float containing the current scale, or 1.0 if scaling is disabled. is_enabled ( ) → bool [source] Return a bool indicating whether this instance is enabled. load_state_dict ( state_dict : dict [ str , Any ] ) → None [source] Load the scaler state. If this instance is disabled, load_state_dict() is a no-op. Parameters : state_dict ( dict ) – scaler state. Should be an object returned from a call to state_dict() . scale ( outputs : TensorBase ) → TensorBase [source] scale ( outputs : list [ TensorBase ] ) → list [ TensorBase ] scale ( outputs : tuple [ TensorBase , ... ] ) → tuple [ TensorBase , ... ] scale ( outputs : Iterable [ tensorplay.Tensor ] ) → Iterable [ tensorplay.Tensor ] Multiplies (‘scales’) a tensor or list of tensors by the scale factor. Returns scaled outputs. If this instance of GradScaler is not enabled, outputs are returned unmodified. Parameters : outputs ( Tensor or iterable of Tensors ) – Outputs to scale. set_backoff_factor ( new_factor : float ) → None [source] Set a new scale backoff factor. Parameters : new_scale ( float ) – Value to use as the new scale backoff factor. set_growth_factor ( new_factor : float ) → None [source] Set a new scale growth factor. Parameters : new_scale ( float ) – Value to use as the new scale growth factor. set_growth_interval ( new_interval : int ) → None [source] Set a new growth interval. Parameters : new_interval ( int ) – Value to use as the new growth interval. state_dict ( ) → dict [ str , Any ] [source] Return the state of the scaler as a dict . It contains five entries: "scale" - a Python float containing the current scale "growth_factor" - a Python float containing the current growth factor "backoff_factor" - a Python float containing the current backoff factor "growth_interval" - a Python int containing the current growth interval "_growth_tracker" - a Python int containing the number of recent consecutive unskipped steps. If this instance is not enabled, returns an empty dict. Note If you wish to checkpoint the scaler’s state after a particular iteration, state_dict() should be called after update() . step ( optimizer : Optimizer , * args : Any , ** kwargs : Any ) → Any [source] Invoke unscale_(optimizer) followed by parameter update, if gradients are not infs/NaN. step() carries out the following two operations: Internally invokes unscale_(optimizer) (unless unscale_() was explicitly called for optimizer earlier in the iteration). As part of the unscale_() , gradients are checked for infs/NaNs. If no inf/NaN gradients are found, invokes optimizer.step() using the unscaled gradients. Otherwise, optimizer.step() is skipped to avoid corrupting the params. *args and **kwargs are forwarded to optimizer.step() . Returns the return value of optimizer.step(*args, **kwargs) . Parameters : optimizer ( tensorplay.optim.Optimizer ) – Optimizer that applies the gradients. args – Any arguments. kwargs – Any keyword arguments. Warning Closure use is not currently supported. unscale_ ( optimizer : Optimizer ) → None [source] Divides (“unscales”) the optimizer’s gradient tensors by the scale factor. unscale_() is optional, serving cases where you need to modify or inspect gradients between the backward pass(es) and step() . If unscale_() is not called explicitly, gradients will be unscaled automatically during step() . Simple example, using unscale_() to enable clipping of unscaled gradients: ... scaler . scale ( loss ) . backward () scaler . unscale_ ( optimizer ) tensorplay . nn . utils . clip_grad_norm_ ( model . parameters (), max_norm ) scaler . step ( optimizer ) scaler . update () Parameters : optimizer ( tensorplay.optim.Optimizer ) – Optimizer that owns the gradients to be unscaled. Warning unscale_() should only be called once per optimizer per step() call, and only after all gradients for that optimizer’s assigned parameters have been accumulated. Calling unscale_() twice for a given optimizer between each step() triggers a RuntimeError. update ( new_scale : float | TensorBase | None = None ) → None [source] Update the scale factor. If any optimizer steps were skipped the scale is multiplied by backoff_factor to reduce it. If growth_interval unskipped iterations occurred consecutively, the scale is multiplied by growth_factor to increase it. Passing new_scale sets the new scale value manually. ( new_scale is not used directly, it’s used to fill GradScaler’s internal scale tensor. So if new_scale was a tensor, later in-place changes to that tensor will not further affect the scale GradScaler uses internally.) Parameters : new_scale (float or tensorplay.Tensor , optional, default=None) – New scale factor. Warning update() should only be called at the end of the iteration, after scaler.step(optimizer) has been invoked for all optimizers used this iteration. # tensorplay.in_parallel_region URL: https://www.tensorplay.cn/docs/generated/tensorplay.in_parallel_region.html tensorplay.in_parallel_region tensorplay. in_parallel_region ( ) → bool Checks whether the code runs in a parallel region # tensorplay.index_fill_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.index_fill_.html tensorplay.index_fill_ tensorplay. index_fill_ ( input , dim , index , value ) [source] # tensorplay.index_fill URL: https://www.tensorplay.cn/docs/generated/tensorplay.index_fill.html tensorplay.index_fill tensorplay. index_fill ( input , dim , index , value ) [source] # tensorplay.index_put URL: https://www.tensorplay.cn/docs/generated/tensorplay.index_put.html tensorplay.index_put tensorplay. index_put ( input , indices , values , accumulate = False ) [source] # tensorplay.initial_seed URL: https://www.tensorplay.cn/docs/generated/tensorplay.initial_seed.html tensorplay.initial_seed tensorplay. initial_seed ( ) → int # tensorplay.inner_backward_other URL: https://www.tensorplay.cn/docs/generated/tensorplay.inner_backward_other.html tensorplay.inner_backward_other tensorplay. inner_backward_other ( grad_output , input , other ) [source] # Layout URL: https://www.tensorplay.cn/docs/generated/tensorplay.Layout.html Layout class tensorplay. Layout ( value ) [source] Sparse/dense storage layout tags (mirrors torch.layout loosely). tensor.layout returns one of these values; dense tensors report STRIDED. as_integer_ratio ( ) Return integer ratio. Return a pair of integers, whose ratio is exactly equal to the original int and with a positive denominator. >>> ( 10 ) . as_integer_ratio () (10, 1) >>> ( - 10 ) . as_integer_ratio () (-10, 1) >>> ( 0 ) . as_integer_ratio () (0, 1) bit_count ( ) Number of ones in the binary representation of the absolute value of self. Also known as the population count. >>> bin ( 13 ) '0b1101' >>> ( 13 ) . bit_count () 3 bit_length ( ) Number of bits necessary to represent self in binary. >>> bin ( 37 ) '0b100101' >>> ( 37 ) . bit_length () 6 conjugate ( ) Returns self, the complex conjugate of any int. denominator the denominator of a rational number in lowest terms from_bytes ( byteorder , * , signed = False ) Return the integer represented by the given array of bytes. bytes Holds the array of bytes to convert. The argument must either support the buffer protocol or be an iterable object producing bytes. Bytes and bytearray are examples of built-in objects that support the buffer protocol. byteorder The byte order used to represent the integer. If byteorder is ‘big’, the most significant byte is at the beginning of the byte array. If byteorder is ‘little’, the most significant byte is at the end of the byte array. To request the native byte order of the host system, use ` sys.byteorder’ as the byte order value. signed Indicates whether two’s complement is used to represent the integer. imag the imaginary part of a complex number numerator the numerator of a rational number in lowest terms real the real part of a complex number to_bytes ( length , byteorder , * , signed = False ) Return an array of bytes representing an integer. length Length of bytes object to use. An OverflowError is raised if the integer is not representable with the given number of bytes. byteorder The byte order used to represent the integer. If byteorder is ‘big’, the most significant byte is at the beginning of the byte array. If byteorder is ‘little’, the most significant byte is at the end of the byte array. To request the native byte order of the host system, use ` sys.byteorder’ as the byte order value. signed Determines whether two’s complement is used to represent the integer. If signed is False and a negative integer is given, an OverflowError is raised. # tensorplay.linalg.det URL: https://www.tensorplay.cn/docs/generated/tensorplay.linalg.det.html tensorplay.linalg.det tensorplay.linalg. det ( A ) → Tensor [source] # tensorplay.linalg.eig URL: https://www.tensorplay.cn/docs/generated/tensorplay.linalg.eig.html tensorplay.linalg.eig tensorplay.linalg. eig ( A) -> (Tensor eigenvalues , Tensor eigenvectors ) [source] # tensorplay.linalg.eigh URL: https://www.tensorplay.cn/docs/generated/tensorplay.linalg.eigh.html tensorplay.linalg.eigh tensorplay.linalg. eigh ( A , UPLO='L') -> (Tensor eigenvalues , Tensor eigenvectors ) [source] # tensorplay.linalg.eigvals URL: https://www.tensorplay.cn/docs/generated/tensorplay.linalg.eigvals.html tensorplay.linalg.eigvals tensorplay.linalg. eigvals ( A ) → Tensor [source] # tensorplay.linalg.eigvalsh URL: https://www.tensorplay.cn/docs/generated/tensorplay.linalg.eigvalsh.html tensorplay.linalg.eigvalsh tensorplay.linalg. eigvalsh ( A , UPLO = 'L' ) → Tensor [source] # tensorplay.linalg.householder_product URL: https://www.tensorplay.cn/docs/generated/tensorplay.linalg.householder_product.html tensorplay.linalg.householder_product tensorplay.linalg. householder_product ( ) # tensorplay.linalg.solve_triangular URL: https://www.tensorplay.cn/docs/generated/tensorplay.linalg.solve_triangular.html tensorplay.linalg.solve_triangular tensorplay.linalg. solve_triangular ( ) # tensorplay.linalg.svd URL: https://www.tensorplay.cn/docs/generated/tensorplay.linalg.svd.html tensorplay.linalg.svd tensorplay.linalg. svd ( A , full_matrices=True , * , driver=None) -> (Tensor U , Tensor S , Tensor Vh ) [source] # tensorplay.linalg.svdvals URL: https://www.tensorplay.cn/docs/generated/tensorplay.linalg.svdvals.html tensorplay.linalg.svdvals tensorplay.linalg. svdvals ( A , * , driver = None ) → Tensor [source] # tensorplay.linalg.tensorinv URL: https://www.tensorplay.cn/docs/generated/tensorplay.linalg.tensorinv.html tensorplay.linalg.tensorinv tensorplay.linalg. tensorinv ( A , ind = 2 ) → Tensor [source] # tensorplay.linalg.tensorsolve URL: https://www.tensorplay.cn/docs/generated/tensorplay.linalg.tensorsolve.html tensorplay.linalg.tensorsolve tensorplay.linalg. tensorsolve ( A , B , dims = None ) → Tensor [source] Solves A X = B where A and B are (tuples of) matrices interpreted as a single square system over the trailing dimensions. # tensorplay.linalg.vander URL: https://www.tensorplay.cn/docs/generated/tensorplay.linalg.vander.html tensorplay.linalg.vander tensorplay.linalg. vander ( x , N = None ) → Tensor [source] # tensorplay.linalg.vecdot URL: https://www.tensorplay.cn/docs/generated/tensorplay.linalg.vecdot.html tensorplay.linalg.vecdot tensorplay.linalg. vecdot ( x , y , * , dim = -1 ) → Tensor [source] # tensorplay.linalg.vector_norm URL: https://www.tensorplay.cn/docs/generated/tensorplay.linalg.vector_norm.html tensorplay.linalg.vector_norm tensorplay.linalg. vector_norm ( x , ord = 2 , dim = None , keepdim = False ) → Tensor [source] # MemoryFormat URL: https://www.tensorplay.cn/docs/generated/tensorplay.MemoryFormat.html MemoryFormat class tensorplay. MemoryFormat ( value ) [source] Tensor memory layout, mirroring torch.MemoryFormat. as_integer_ratio ( ) Return integer ratio. Return a pair of integers, whose ratio is exactly equal to the original int and with a positive denominator. >>> ( 10 ) . as_integer_ratio () (10, 1) >>> ( - 10 ) . as_integer_ratio () (-10, 1) >>> ( 0 ) . as_integer_ratio () (0, 1) bit_count ( ) Number of ones in the binary representation of the absolute value of self. Also known as the population count. >>> bin ( 13 ) '0b1101' >>> ( 13 ) . bit_count () 3 bit_length ( ) Number of bits necessary to represent self in binary. >>> bin ( 37 ) '0b100101' >>> ( 37 ) . bit_length () 6 conjugate ( ) Returns self, the complex conjugate of any int. denominator the denominator of a rational number in lowest terms from_bytes ( byteorder , * , signed = False ) Return the integer represented by the given array of bytes. bytes Holds the array of bytes to convert. The argument must either support the buffer protocol or be an iterable object producing bytes. Bytes and bytearray are examples of built-in objects that support the buffer protocol. byteorder The byte order used to represent the integer. If byteorder is ‘big’, the most significant byte is at the beginning of the byte array. If byteorder is ‘little’, the most significant byte is at the end of the byte array. To request the native byte order of the host system, use ` sys.byteorder’ as the byte order value. signed Indicates whether two’s complement is used to represent the integer. imag the imaginary part of a complex number numerator the numerator of a rational number in lowest terms real the real part of a complex number to_bytes ( length , byteorder , * , signed = False ) Return an array of bytes representing an integer. length Length of bytes object to use. An OverflowError is raised if the integer is not representable with the given number of bytes. byteorder The byte order used to represent the integer. If byteorder is ‘big’, the most significant byte is at the beginning of the byte array. If byteorder is ‘little’, the most significant byte is at the end of the byte array. To request the native byte order of the host system, use ` sys.byteorder’ as the byte order value. signed Determines whether two’s complement is used to represent the integer. If signed is False and a negative integer is given, an OverflowError is raised. # tensorplay.native_dropout URL: https://www.tensorplay.cn/docs/generated/tensorplay.native_dropout.html tensorplay.native_dropout tensorplay. native_dropout ( input , p ) [source] # tensorplay.nll_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nll_loss.html tensorplay.nll_loss tensorplay. nll_loss ( input , target , weight = None , reduction = 1 , ignore_index = -100 ) [source] # Buffer URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.Buffer.html Buffer class tensorplay.nn. Buffer ( data = None , persistent = True ) [source] A kind of Tensor that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers are Tensor subclasses, that have a very special property when used with Module s – when they’re assigned as Module attributes they are automatically added to the list of its buffers, and will appear e.g. in buffers() iterator. Assigning a Tensor doesn’t have such effect. One can still assign a Tensor as explicitly by using the register_buffer() function. Parameters : data ( Tensor ) – buffer tensor. persistent ( bool , optional ) – whether the buffer is part of the module’s state_dict . Default: True abs ( Tensor self ) → Tensor abs_ ( Tensor(a!) self) -> Tensor(a! ) acos ( Tensor self ) → Tensor acosh ( Tensor self ) → Tensor add ( ) add_ ( ) add_ .Tensor(Tensor(a!) self, Tensor other, * , Scalar alpha=1) -> Tensor(a!) | add_ .Scalar(Tensor(a!) self, Scalar other, Scalar alpha=1) -> Tensor(a!) addbmm ( Tensor self , Tensor batch1 , Tensor batch2 , * , Scalar beta=1 , Scalar alpha=1 ) → Tensor addcdiv ( Tensor self , Tensor tensor1 , Tensor tensor2 , * , Scalar value=1 ) → Tensor addcdiv_ ( Tensor(a!) self , Tensor tensor1 , Tensor tensor2 , * , Scalar value=1) -> Tensor(a! ) addcmul ( Tensor self , Tensor tensor1 , Tensor tensor2 , * , Scalar value=1 ) → Tensor addcmul_ ( Tensor(a!) self , Tensor tensor1 , Tensor tensor2 , * , Scalar value=1) -> Tensor(a! ) addmv ( Tensor self , Tensor mat , Tensor vec , * , Scalar beta=1 , Scalar alpha=1 ) → Tensor addr ( Tensor self , Tensor vec1 , Tensor vec2 , * , Scalar beta=1 , Scalar alpha=1 ) → Tensor airy_ai ( Tensor self ) → Tensor all ( Tensor self) -> Tensor | all.dim(Tensor self , int[] dim , bool keepdim=false ) → Tensor allclose ( Tensor self , Tensor other , float rtol=1e-05 , float atol=1e-08 , bool equal_nan=False ) → bool angle ( Tensor self ) → Tensor any ( Tensor self) -> Tensor | any.dim(Tensor self , int[] dim , bool keepdim=false ) → Tensor argmax ( Tensor self , int? dim=None , bool keepdim=False ) → Tensor argmin ( Tensor self , int? dim=None , bool keepdim=False ) → Tensor argsort ( Tensor self , int dim=-1 , bool descending=False ) → Tensor argwhere ( Tensor self ) → Tensor as_strided ( self : tensorplay._C.TensorBase , size : collections.abc.Sequence [ SupportsInt | SupportsIndex ] , stride : collections.abc.Sequence [ SupportsInt | SupportsIndex ] , storage_offset : SupportsInt | SupportsIndex | None = None ) → tensorplay._C.TensorBase asin ( Tensor self ) → Tensor asinh ( Tensor self ) → Tensor atan ( Tensor self ) → Tensor atan2 ( Tensor self , Tensor other ) → Tensor atanh ( Tensor self ) → Tensor backward ( self : tensorplay._C.TensorBase , gradient : tensorplay._C.TensorBase | None = None , retain_graph : bool | None = None , create_graph : bool = False ) → None bernoulli ( Tensor self ) → Tensor bernoulli_ ( Tensor(a!) self) -> Tensor(a! ) bessel_j0 ( Tensor self ) → Tensor bessel_j1 ( Tensor self ) → Tensor bessel_y0 ( Tensor self ) → Tensor bessel_y1 ( Tensor self ) → Tensor bincount ( Tensor self , Tensor? weights=None , int minlength=0 ) → Tensor bitwise_and ( ) bitwise_left_shift ( ) bitwise_not ( Tensor self ) → Tensor bitwise_or ( ) bitwise_right_shift ( ) bitwise_xor ( ) bmm ( Tensor self , Tensor mat2 ) → Tensor broadcast_to ( Tensor self , int[] size ) → Tensor cauchy_ ( Tensor(a!) self , float median=0.0 , float sigma=1.0) -> Tensor(a! ) ceil ( Tensor self ) → Tensor celu ( Tensor self , Scalar alpha=1.0 ) → Tensor channel_shuffle ( Tensor self , int groups ) → Tensor cholesky ( Tensor self , bool upper=False ) → Tensor cholesky_inverse ( Tensor self , bool upper=False ) → Tensor cholesky_solve ( Tensor self , Tensor input2 , bool upper=False ) → Tensor chunk ( Tensor self , int chunks , int dim=0 ) → Tensor[] clamp ( Tensor self , Scalar? min=None , Scalar? max=None ) → Tensor clamp_ ( Tensor(a!) self , Scalar? min=None , Scalar? max=None) -> Tensor(a! ) clip ( Tensor self , Scalar? min=None , Scalar? max=None ) → Tensor clone ( Tensor self , * , MemoryFormat? memory_format=None ) → Tensor coalesce ( self : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase col_indices ( self : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase conj ( Tensor self ) → Tensor contiguous ( Tensor self , * , MemoryFormat memory_format=Contiguous ) → Tensor copy_ ( Tensor(a!) self , Tensor src , bool non_blocking=False) -> Tensor(a! ) cos ( Tensor self ) → Tensor cosh ( Tensor self ) → Tensor cpu ( ) Returns a copy of this object in CPU memory. If this object is already in CPU memory, then no copy is performed and the original object is returned. crow_indices ( self : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase cuda ( device = None , non_blocking = False ) Returns a copy of this object in CUDA memory. If this object is already in CUDA memory and on the correct device, then no copy is performed and the original object is returned. cummax ( Tensor self , int dim) -> (Tensor values , Tensor indices ) cummin ( Tensor self , int dim) -> (Tensor values , Tensor indices ) cumprod ( Tensor self , int dim , ScalarType? dtype=None ) → Tensor cumsum ( Tensor self , int dim=0 , ScalarType? dtype=None ) → Tensor data_ptr ( self : tensorplay._C.TensorBase ) → int deg2rad ( Tensor self ) → Tensor dense_dim ( self : tensorplay._C.TensorBase ) → int dequantize_per_channel ( Tensor self , Tensor scales , Tensor zero_points , int axis=0 ) → Tensor dequantize_per_tensor ( Tensor self , float scale , int zero_point ) → Tensor detach ( self : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase detach_ ( self : object ) → object diag ( Tensor self , int diagonal=0 ) → Tensor diag_embed ( Tensor self , int offset=0 , int dim1=0 , int dim2=1 ) → Tensor diagonal ( Tensor self , int offset=0 , int dim1=0 , int dim2=1 ) → Tensor diagonal_scatter ( Tensor self , Tensor src , int offset=0 , int dim1=0 , int dim2=1 ) → Tensor digamma ( Tensor self ) → Tensor dim ( self : tensorplay._C.TensorBase ) → int dist ( Tensor self , Tensor other , Scalar p=2 ) → Tensor div ( ) div_ ( ) div_ .Tensor(Tensor(a!) self, Tensor other) -> Tensor(a!) | div_ .Scalar(Tensor(a!) self, Scalar other) -> Tensor(a!) dot ( Tensor self , Tensor tensor ) → Tensor dsplit ( ) element_size ( self : tensorplay._C.TensorBase ) → int elu ( Tensor self , Scalar alpha=1 , Scalar scale=1 , Scalar input_scale=1 ) → Tensor eq ( ) equal ( Tensor self , Tensor other ) → bool erf ( Tensor self ) → Tensor erfc ( Tensor self ) → Tensor erfinv ( Tensor self ) → Tensor exp ( Tensor self ) → Tensor exp2 ( Tensor self ) → Tensor expand_as ( Tensor self , Tensor other ) → Tensor expm1 ( Tensor self ) → Tensor exponential_ ( Tensor(a!) self , float lambd=1.0) -> Tensor(a! ) fill_ ( ) fill_ .Scalar(Tensor(a!) self, Scalar value) -> Tensor(a!) fix ( Tensor self ) → Tensor flatten ( start_dim = 0 , end_dim = -1 ) Flattens a contiguous range of dims. floor ( Tensor self ) → Tensor frac ( Tensor self ) → Tensor gather ( Tensor self , int dim , Tensor index ) → Tensor gcd ( Tensor self , Tensor other ) → Tensor ge ( ) gelu ( Tensor self , str approximate="none" ) → Tensor geometric_ ( Tensor(a!) self , float p) -> Tensor(a! ) glu ( Tensor self , int dim=-1 ) → Tensor greater ( Tensor self , Tensor other ) → Tensor greater_equal ( Tensor self , Tensor other ) → Tensor gt ( ) hardshrink ( Tensor self , Scalar lambd=0.5 ) → Tensor hardsigmoid ( Tensor self ) → Tensor hardswish ( Tensor self ) → Tensor hardtanh ( Tensor self , Scalar min_val=-1 , Scalar max_val=1 ) → Tensor heaviside ( Tensor self , Tensor values ) → Tensor hsplit ( ) hypot ( Tensor self , Tensor other ) → Tensor i0e ( Tensor self ) → Tensor i1 ( Tensor self ) → Tensor i1e ( Tensor self ) → Tensor imag ( Tensor self ) → Tensor index_copy ( Tensor self , int dim , Tensor index , Tensor source ) → Tensor index_fill ( ) index_fill_ ( ) index_fill_ .Scalar(Tensor(a!) self, int dim, Tensor index, Scalar value) -> Tensor(a!) | index_fill_ .Tensor(Tensor(a!) self, int dim, Tensor index, Tensor value) -> Tensor(a!) index_put ( Tensor self , Tensor[] indices , Tensor values , bool accumulate=False ) → Tensor index_put_ ( Tensor(a!) self , Tensor[] indices , Tensor values , bool accumulate=False) -> Tensor(a! ) index_select ( Tensor self , int dim , Tensor index ) → Tensor inner ( Tensor self , Tensor other ) → Tensor is_channels_last ( self : tensorplay._C.TensorBase ) → bool is_channels_last_2d ( self : tensorplay._C.TensorBase ) → bool is_channels_last_3d ( self : tensorplay._C.TensorBase ) → bool is_coalesced ( self : tensorplay._C.TensorBase ) → bool is_complex ( self : tensorplay._C.TensorBase ) → bool is_contiguous ( * args , ** kwargs ) Overloaded function. is_contiguous(self: tensorplay._C.TensorBase) -> bool is_contiguous(self: tensorplay._C.TensorBase, memory_format: typing.SupportsInt | typing.SupportsIndex) -> bool is_floating_point ( self : tensorplay._C.TensorBase ) → bool is_pinned ( self : tensorplay._C.TensorBase ) → bool is_shared ( self : object ) → bool is_sparse_csr ( self : tensorplay._C.TensorBase ) → bool isclose ( Tensor self , Tensor other , float rtol=1e-05 , float atol=1e-08 , bool equal_nan=False ) → Tensor isfinite ( Tensor self ) → Tensor isinf ( Tensor self ) → Tensor isnan ( Tensor self ) → Tensor isneginf ( Tensor self ) → Tensor isposinf ( Tensor self ) → Tensor isreal ( Tensor self ) → Tensor itemsize ( self : tensorplay._C.TensorBase ) → int kthvalue ( Tensor self , int k , int dim=-1 , bool keepdim=False) -> (Tensor values , Tensor indices ) lcm ( Tensor self , Tensor other ) → Tensor le ( ) leaky_relu ( Tensor self , Scalar negative_slope=0.01 ) → Tensor lerp ( Tensor self , Tensor end , Scalar weight) -> Tensor | lerp.Tensor(Tensor self , Tensor end , Tensor weight ) → Tensor lerp_ ( ) lerp_ .Scalar(Tensor(a!) self, Tensor end, Scalar weight) -> Tensor(a!) | lerp_ .Tensor(Tensor(a!) self, Tensor end, Tensor weight) -> Tensor(a!) less ( Tensor self , Tensor other ) → Tensor less_equal ( Tensor self , Tensor other ) → Tensor lgamma ( Tensor self ) → Tensor log ( Tensor self ) → Tensor log10 ( Tensor self ) → Tensor log1p ( Tensor self ) → Tensor log2 ( Tensor self ) → Tensor log_normal_ ( Tensor(a!) self , float mean=1.0 , float std=2.0) -> Tensor(a! ) log_softmax ( Tensor self , int dim , ScalarType dtype=Undefined ) → Tensor logaddexp ( Tensor self , Tensor other ) → Tensor logaddexp2 ( Tensor self , Tensor other ) → Tensor logcumsumexp ( Tensor self , int dim , ScalarType? dtype=None ) → Tensor logical_not ( Tensor self ) → Tensor logical_or ( Tensor self , Tensor other ) → Tensor logical_xor ( Tensor self , Tensor other ) → Tensor logit ( Tensor self , Scalar? eps=None ) → Tensor logsumexp ( Tensor self , int dim , bool keepdim=False ) → Tensor lt ( ) masked_fill ( ) masked_fill_ ( ) masked_fill_ .Tensor(Tensor(a!) self, Tensor mask, Tensor value) -> Tensor(a!) | masked_fill_(Tensor(a!) self, Tensor mask, Scalar value) -> Tensor(a!) masked_scatter ( Tensor self , Tensor mask , Tensor source ) → Tensor masked_select ( Tensor self , Tensor mask ) → Tensor matmul ( Tensor self , Tensor other ) → Tensor max ( Tensor self) -> Tensor | max.dim(Tensor self , int[] dim , bool keepdim=false ) → Tensor mean ( Tensor self , * , ScalarType dtype=Undefined) -> Tensor | mean.dim(Tensor self , int[] dim , bool keepdim=false , * , ScalarType dtype=Undefined ) → Tensor median ( Tensor self ) → Tensor memory_format ( self : tensorplay._C.TensorBase ) → int min ( Tensor self) -> Tensor | min.dim(Tensor self , int[] dim , bool keepdim=false ) → Tensor mish ( Tensor self ) → Tensor mm ( Tensor self , Tensor mat2 ) → Tensor mode ( Tensor self , int dim=-1 , bool keepdim=False) -> (Tensor values , Tensor indices ) modified_bessel_i1 ( Tensor self ) → Tensor modified_bessel_k0 ( Tensor self ) → Tensor modified_bessel_k1 ( Tensor self ) → Tensor moveaxis ( ) movedim ( Tensor self , int[] source , int[] destination ) → Tensor msort ( Tensor self ) → Tensor mul ( ) mul_ ( ) mul_ .Tensor(Tensor(a!) self, Tensor other) -> Tensor(a!) | mul_ .Scalar(Tensor(a!) self, Scalar other) -> Tensor(a!) mv ( Tensor self , Tensor vec ) → Tensor nan_to_num ( Tensor self , Scalar nan=0.0 , Scalar? posinf=None , Scalar? neginf=None ) → Tensor nanmean ( Tensor self , int? dim=None , bool keepdim=False , * , ScalarType? dtype=None ) → Tensor nanmedian ( Tensor self ) → Tensor nansum ( Tensor self , int[] dim=[] , bool keepdim=False ) → Tensor narrow ( Tensor self , int dim , int start , int length ) → Tensor ndimension ( ) → int Alias for dim() ne ( ) neg ( Tensor self ) → Tensor neg_ ( Tensor(a!) self) -> Tensor(a! ) negative ( Tensor self ) → Tensor nextafter ( Tensor self , Tensor other ) → Tensor norm ( Tensor self , float p=2.0) -> Tensor | norm.dim(Tensor self , int[] dim , float p=2.0 , bool keepdim=false ) → Tensor normal_ ( Tensor(a!) self , float mean=0.0 , float std=1.0) -> Tensor(a! ) not_equal ( Tensor self , Tensor other ) → Tensor numel ( self : tensorplay._C.TensorBase ) → int numpy ( self : object ) → numpy.ndarray outer ( Tensor self , Tensor vec2 ) → Tensor pdist ( Tensor self , float p=2.0 ) → Tensor pin_memory ( self : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase poisson ( Tensor self ) → Tensor polygamma ( int n , Tensor self ) → Tensor positive ( Tensor self ) → Tensor pow ( ) prelu ( Tensor self , Tensor weight ) → Tensor prod ( Tensor self , * , ScalarType dtype=Undefined) -> Tensor | prod.dim_IntList(Tensor self , int[] dim , bool keepdim=false , * , ScalarType dtype=Undefined ) → Tensor quantize_per_channel ( Tensor self , Tensor scales , Tensor zero_points , int axis=0 ) → Tensor quantize_per_tensor ( Tensor self , float scale , int zero_point , int quant_min=-128 , int quant_max=127 ) → Tensor rad2deg ( Tensor self ) → Tensor random_ ( Tensor(a!) self , int low=0 , int high=0) -> Tensor(a! ) ravel ( Tensor self ) → Tensor reciprocal ( Tensor self ) → Tensor register_hook ( hook ) Registers a backward hook (torch’s Tensor.register_hook ). The hook is called every time a gradient with respect to this tensor is computed. It may modify the gradient by returning a replacement Tensor; returning None leaves the gradient unchanged. Hooks compose in registration order. Returns a RemovableHandle whose remove() method (or context-manager form) unregisters the hook. register_post_accumulate_grad_hook ( hook ) Registers a hook (torch’s Tensor.register_post_accumulate_grad_hook ). The hook runs after the gradient has been accumulated into self.grad . It receives the tensor (the parameter) and its return value is ignored; unlike register_hook() it cannot replace the gradient, but it may modify self.grad in place. Only leaf tensors that require grad and are used in the autograd graph support this hook. Returns a RemovableHandle . relu ( Tensor self ) → Tensor relu6 ( Tensor self ) → Tensor relu_ ( Tensor(a!) self) -> Tensor(a! ) renorm ( Tensor self , Scalar p , int dim , Scalar maxnorm ) → Tensor repeat ( Tensor self , int[] repeats ) → Tensor requires_grad_ ( self : object , requires_grad : bool = True ) → object reshape ( Tensor self , int[] shape ) → Tensor resize_ ( Tensor(a!) self , int[] size) -> Tensor(a! ) retain_grad ( self : tensorplay._C.TensorBase ) → None rot90 ( Tensor self , int k=1 , int[] dims=[] ) → Tensor round ( Tensor self ) → Tensor rsqrt ( Tensor self ) → Tensor rsqrt_ ( Tensor(a!) self) -> Tensor(a! ) scaled_modified_bessel_k0 ( Tensor self ) → Tensor scaled_modified_bessel_k1 ( Tensor self ) → Tensor scatter ( ) scatter_ ( ) scatter_ .src(Tensor(a!) self, int dim, Tensor index, Tensor src) -> Tensor(a!) | scatter_ .value(Tensor(a!) self, int dim, Tensor index, Scalar value) -> Tensor(a!) scatter_add ( Tensor self , int dim , Tensor index , Tensor src ) → Tensor scatter_add_ ( Tensor(a!) self , int dim , Tensor index , Tensor src) -> Tensor(a! ) select ( Tensor self , int dim , int index ) → Tensor select_scatter ( Tensor self , Tensor src , int dim , int index ) → Tensor selu ( Tensor self ) → Tensor sgn ( Tensor self ) → Tensor share_memory_ ( self : object ) → object sigmoid ( Tensor self ) → Tensor sign ( Tensor self ) → Tensor signbit ( Tensor self ) → Tensor silu ( Tensor self ) → Tensor sin ( Tensor self ) → Tensor sinc ( Tensor self ) → Tensor sinh ( Tensor self ) → Tensor size ( * args , ** kwargs ) Overloaded function. size(self: tensorplay._C.TensorBase) -> tensorplay._C.Size size(self: tensorplay._C.TensorBase, arg0: typing.SupportsInt | typing.SupportsIndex) -> int slice ( Tensor self , int dim=0 , int? start=None , int? end=None , int step=1 ) → Tensor slice_scatter ( Tensor self , Tensor src , int dim=0 , int? start=None , int? end=None , int step=1 ) → Tensor softmax ( Tensor self , int dim , ScalarType dtype=Undefined ) → Tensor softplus ( Tensor self , Scalar beta=1 , Scalar threshold=20 ) → Tensor softshrink ( Tensor self , Scalar lambd=0.5 ) → Tensor sort ( Tensor self , int dim=-1 , bool descending=False) -> (Tensor values , Tensor indices ) sparse_dim ( self : tensorplay._C.TensorBase ) → int sparse_mask ( self : tensorplay._C.TensorBase , mask : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase sparse_sum ( Tensor self , int[]? dim=None , ScalarType? dtype=None ) → Tensor spherical_bessel_j0 ( Tensor self ) → Tensor split ( Tensor self , int split_size , int dim=0) -> Tensor[] | split.sizes(Tensor self , int[] split_sizes , int dim=0 ) → Tensor[] split_with_sizes ( Tensor self , int[] split_sizes , int dim=0 ) → Tensor[] sqrt ( Tensor self ) → Tensor sqrt_ ( Tensor(a!) self) -> Tensor(a! ) square ( Tensor self ) → Tensor squeeze ( Tensor self) -> Tensor | squeeze.dim(Tensor self , int dim ) → Tensor std ( Tensor self , int correction=1) -> Tensor | std.dim(Tensor self , int[] dim , int correction=1 , bool keepdim=false ) → Tensor stride ( * args , ** kwargs ) Overloaded function. stride(self: tensorplay._C.TensorBase) -> tuple stride(self: tensorplay._C.TensorBase, arg0: typing.SupportsInt | typing.SupportsIndex) -> int sub ( ) sub_ ( ) sub_ .Tensor(Tensor(a!) self, Tensor other, * , Scalar alpha=1) -> Tensor(a!) | sub_ .Scalar(Tensor(a!) self, Scalar other, Scalar alpha=1) -> Tensor(a!) sum ( Tensor self , * , ScalarType dtype=Undefined) -> Tensor | sum.dim_IntList(Tensor self , int[] dim , bool keepdim=false , * , ScalarType dtype=Undefined ) → Tensor svd ( Tensor self , bool some=True , bool compute_uv=True) -> (Tensor U , Tensor S , Tensor V ) swapaxes ( Tensor self , int axis0 , int axis1 ) → Tensor swapdims ( Tensor self , int dim0 , int dim1 ) → Tensor t ( ) Returns the transpose of the tensor. Aliased to transpose(0, 1) to ensure correct autograd behavior (TransposeBackward). take ( Tensor self , Tensor index ) → Tensor take_along_dim ( Tensor self , Tensor indices , int? dim=None ) → Tensor tan ( Tensor self ) → Tensor tanh ( Tensor self ) → Tensor tensor_split ( ) tile ( Tensor self , int[] dims ) → Tensor to ( * args , ** kwargs ) Overloaded function. to(self: tensorplay._C.TensorBase, dtype: tensorplay._C.DType, non_blocking: bool = False, copy: bool = False) -> tensorplay._C.TensorBase to(self: tensorplay._C.TensorBase, device: tensorplay._C.Device, non_blocking: bool = False, copy: bool = False) -> tensorplay._C.TensorBase to(self: tensorplay._C.TensorBase, device: tensorplay._C.Device, dtype: tensorplay._C.DType, non_blocking: bool = False, copy: bool = False) -> tensorplay._C.TensorBase to_dense ( Tensor self ) → Tensor to_sparse ( Tensor self ) → Tensor to_sparse_csr ( Tensor self ) → Tensor tolist ( self : tensorplay._C.TensorBase ) → object trace ( Tensor self ) → Tensor transpose ( Tensor self , int dim0 , int dim1 ) → Tensor triangular_solve ( Tensor self , Tensor A , bool upper=False , bool transpose=False , bool unitriangular=False) -> (Tensor solution , Tensor cloned_coefficient ) tril ( Tensor self , int diagonal=0 ) → Tensor triu ( Tensor self , int diagonal=0 ) → Tensor trunc ( Tensor self ) → Tensor type ( dtype = None , non_blocking = False , ** kwargs ) Returns the type if dtype is not provided, else casts this object to the specified type. unbind ( Tensor self , int dim=0 ) → Tensor[] unflatten ( dim , sizes ) Expands a dimension of the input tensor over multiple dimensions. unfold ( dimension , size , step ) Returns a view of the original tensor which contains all slices of size size from self in the dimension dimension , stepping by step (torch’s Tensor.unfold ). Port of aten/src/ATen/native/TensorShape.cpp : the view appends a new trailing dimension of length size and re-strides dimension by step . uniform_ ( Tensor(a!) self , float from=0.0 , float to=1.0) -> Tensor(a! ) unsqueeze ( Tensor self , int dim ) → Tensor values ( self : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase var ( Tensor self , int correction=1) -> Tensor | var.dim(Tensor self , int[] dim , int correction=1 , bool keepdim=false ) → Tensor vdot ( Tensor self , Tensor other ) → Tensor view ( Tensor self , int[] shape ) → Tensor vsplit ( ) zero_ ( Tensor(a!) self) -> Tensor(a! ) zeta ( Tensor self , Tensor other ) → Tensor # DepthwiseConv2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.DepthwiseConv2d.html DepthwiseConv2d class tensorplay.nn. DepthwiseConv2d ( in_channels , out_channels , kernel_size , stride = 1 , padding = 0 , dilation = 1 , bias = True , padding_mode = 'zeros' , device = None , dtype = None ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # tensorplay.nn.factory_kwargs URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.factory_kwargs.html tensorplay.nn.factory_kwargs tensorplay.nn. factory_kwargs ( kwargs ) [source] Return a canonicalized dict of factory kwargs. Given kwargs, returns a canonicalized dict of factory kwargs that can be directly passed to factory functions like torch.empty, or errors if unrecognized kwargs are present. This function makes it simple to write code like this: class MyModule ( nn . Module ): def __init__ ( self , ** kwargs ): factory_kwargs = torch . nn . factory_kwargs ( kwargs ) self . weight = Parameter ( torch . empty ( 10 , ** factory_kwargs )) Why should you use this function instead of just passing kwargs along directly? 1. This function does error validation, so if there are unexpected kwargs we will immediately report an error, instead of deferring it to the factory call 2. This function supports a special factory_kwargs argument, which can be used to explicitly specify a kwarg to be used for factory functions, in the event one of the factory kwargs conflicts with an already existing argument in the signature (e.g. in the signature def f(dtype, **kwargs) , you can specify dtype for factory functions, as distinct from the dtype argument, by saying f(dtype1, factory_kwargs={"dtype": dtype2}) ) # tensorplay.nn.functional.adaptive_avg_pool1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.adaptive_avg_pool1d.html tensorplay.nn.functional.adaptive_avg_pool1d tensorplay.nn.functional. adaptive_avg_pool1d ( input , output_size ) → Tensor [source] # tensorplay.nn.functional.adaptive_avg_pool2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.adaptive_avg_pool2d.html tensorplay.nn.functional.adaptive_avg_pool2d tensorplay.nn.functional. adaptive_avg_pool2d ( input , output_size ) [source] # tensorplay.nn.functional.adaptive_avg_pool3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.adaptive_avg_pool3d.html tensorplay.nn.functional.adaptive_avg_pool3d tensorplay.nn.functional. adaptive_avg_pool3d ( input : TensorBase , output_size ) → TensorBase [source] Apply a 3D adaptive average pooling over an input signal. # tensorplay.nn.functional.adaptive_max_pool1d_with_indices URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.adaptive_max_pool1d_with_indices.html tensorplay.nn.functional.adaptive_max_pool1d_with_indices tensorplay.nn.functional. adaptive_max_pool1d_with_indices ( input : TensorBase , output_size , return_indices : bool = True ) [source] Applies a 1D adaptive max pooling over an input signal, returning (output, indices) . See AdaptiveMaxPool1d for details. # tensorplay.nn.functional.adaptive_max_pool1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.adaptive_max_pool1d.html tensorplay.nn.functional.adaptive_max_pool1d tensorplay.nn.functional. adaptive_max_pool1d ( input , output_size ) → Tensor [source] # tensorplay.nn.functional.adaptive_max_pool2d_with_indices URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.adaptive_max_pool2d_with_indices.html tensorplay.nn.functional.adaptive_max_pool2d_with_indices tensorplay.nn.functional. adaptive_max_pool2d_with_indices ( input : TensorBase , output_size , return_indices : bool = True ) [source] Applies a 2D adaptive max pooling over an input signal composed of several input planes, returning (output, indices) . See AdaptiveMaxPool2d for details. # tensorplay.nn.functional.adaptive_max_pool2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.adaptive_max_pool2d.html tensorplay.nn.functional.adaptive_max_pool2d tensorplay.nn.functional. adaptive_max_pool2d ( input , output_size ) [source] # tensorplay.nn.functional.adaptive_max_pool3d_with_indices URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.adaptive_max_pool3d_with_indices.html tensorplay.nn.functional.adaptive_max_pool3d_with_indices tensorplay.nn.functional. adaptive_max_pool3d_with_indices ( input : TensorBase , output_size , return_indices : bool = True ) [source] Applies a 3D adaptive max pooling over an input signal, returning (output, indices) . See AdaptiveMaxPool3d for details. # tensorplay.nn.functional.affine_grid URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.affine_grid.html tensorplay.nn.functional.affine_grid tensorplay.nn.functional. affine_grid ( theta : TensorBase , size , align_corners = None ) → TensorBase [source] Generate 2D or 3D flow field (sampling grid), given a batch of affine matrices theta . Port of aten/src/ATen/native/AffineGridGenerator.cpp. # tensorplay.nn.functional.alpha_dropout URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.alpha_dropout.html tensorplay.nn.functional.alpha_dropout tensorplay.nn.functional. alpha_dropout ( input , p = 0.5 , training = True , inplace = False ) [source] # tensorplay.nn.functional.avg_pool1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.avg_pool1d.html tensorplay.nn.functional.avg_pool1d tensorplay.nn.functional. avg_pool1d ( input : TensorBase , kernel_size , stride = None , padding = 0 , ceil_mode : bool = False , count_include_pad : bool = True , divisor_override = None ) → TensorBase [source] avg_pool1d(input, kernel_size, stride=None, padding=0, ceil_mode=False, count_include_pad=True, divisor_override=None) -> Tensor Applies a 1D average pooling over an input signal composed of several input planes. Input shape (N, C, L) or unbatched (C, L) . # tensorplay.nn.functional.avg_pool2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.avg_pool2d.html tensorplay.nn.functional.avg_pool2d tensorplay.nn.functional. avg_pool2d ( input , kernel_size , stride = None , padding = 0 , ceil_mode = False , count_include_pad = True , divisor_override = None ) [source] # tensorplay.nn.functional.avg_pool3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.avg_pool3d.html tensorplay.nn.functional.avg_pool3d tensorplay.nn.functional. avg_pool3d ( input : TensorBase , kernel_size , stride = None , padding = 0 , ceil_mode : bool = False , count_include_pad : bool = True , divisor_override = None ) → TensorBase [source] avg_pool3d(input, kernel_size, stride=None, padding=0, ceil_mode=False, count_include_pad=True, divisor_override=None) -> Tensor Applies a 3D average pooling over an input signal composed of several input planes. Input shape (N, C, D, H, W) or unbatched (C, D, H, W) . # tensorplay.nn.functional.batch_norm URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.batch_norm.html tensorplay.nn.functional.batch_norm tensorplay.nn.functional. batch_norm ( input , running_mean = None , running_var = None , weight = None , bias = None , training = False , momentum = 0.1 , eps = 1e-05 ) [source] # tensorplay.nn.functional.bilinear URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.bilinear.html tensorplay.nn.functional.bilinear tensorplay.nn.functional. bilinear ( input1 , input2 , weight , bias = None ) [source] # tensorplay.nn.functional.binary_cross_entropy URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.binary_cross_entropy.html tensorplay.nn.functional.binary_cross_entropy tensorplay.nn.functional. binary_cross_entropy ( input : TensorBase , target : TensorBase , weight : TensorBase | None = None , size_average = None , reduce = None , reduction : str = 'mean' ) → TensorBase [source] Compute Binary Cross Entropy between the target and input probabilities. See BCELoss for details. # tensorplay.nn.functional.channel_shuffle URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.channel_shuffle.html tensorplay.nn.functional.channel_shuffle tensorplay.nn.functional. channel_shuffle ( input : TensorBase , groups : int ) → TensorBase [source] Divide the channels in a tensor into g groups and rearrange them as in ShuffleNet: (*, C, H, W) -> (*, C, H, W) with channels interleaved across groups. # tensorplay.nn.functional.conv_tbc URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.conv_tbc.html tensorplay.nn.functional.conv_tbc tensorplay.nn.functional. conv_tbc ( input , weight , bias = None , pad = 0 ) [source] Applies a 1D convolution over an input of shape (T, B, C) along the time dimension (torch.conv_tbc). Expressed as a grouped (depthwise) conv1d: every channel is convolved with its own length-k filter. # tensorplay.nn.functional.conv_transpose1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.conv_transpose1d.html tensorplay.nn.functional.conv_transpose1d tensorplay.nn.functional. conv_transpose1d ( input , weight , bias = None , stride = 1 , padding = 0 , output_padding = 0 , groups = 1 , dilation = 1 ) [source] # tensorplay.nn.functional.conv_transpose2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.conv_transpose2d.html tensorplay.nn.functional.conv_transpose2d tensorplay.nn.functional. conv_transpose2d ( input , weight , bias = None , stride = 1 , padding = 0 , output_padding = 0 , groups = 1 , dilation = 1 ) [source] # tensorplay.nn.functional.conv_transpose3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.conv_transpose3d.html tensorplay.nn.functional.conv_transpose3d tensorplay.nn.functional. conv_transpose3d ( input , weight , bias = None , stride = 1 , padding = 0 , output_padding = 0 , groups = 1 , dilation = 1 ) [source] # tensorplay.nn.functional.conv1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.conv1d.html tensorplay.nn.functional.conv1d tensorplay.nn.functional. conv1d ( input , weight , bias = None , stride = 1 , padding = 0 , dilation = 1 , groups = 1 ) [source] Applies a 1D convolution over an input signal composed of several input planes. See Conv1d for details and output shape. Parameters : input – input tensor of shape \((\text{minibatch} , \text{in\_channels} , iW)\) weight – filters of shape \((\text{out\_channels} , \frac{\text{in\_channels}}{\text{groups}} , kW)\) bias – optional bias of shape \((\text{out\_channels})\) . Default: None stride – the stride of the convolving kernel. Can be a single number or a one-element tuple (sW,) . Default: 1 padding – implicit paddings on both sides of the input. Can be a single number or a one-element tuple (padW,) . Default: 0 dilation – the spacing between kernel elements. Can be a single number or a one-element tuple (dW,) . Default: 1 groups – split input into groups, \(\text{in\_channels}\) should be divisible by the number of groups. Default: 1 Examples: >>> inputs = tp . randn ( 33 , 16 , 30 ) >>> filters = tp . randn ( 20 , 16 , 5 ) >>> F . conv1d ( inputs , filters ) # tensorplay.nn.functional.conv2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.conv2d.html tensorplay.nn.functional.conv2d tensorplay.nn.functional. conv2d ( input , weight , bias = None , stride = 1 , padding = 0 , dilation = 1 , groups = 1 ) [source] Applies a 2D convolution over an input image composed of several input planes. See Conv2d for details and output shape. Parameters : input – input tensor of shape \((\text{minibatch} , \text{in\_channels} , iH , iW)\) weight – filters of shape \((\text{out\_channels} , \frac{\text{in\_channels}}{\text{groups}} , kH , kW)\) bias – optional bias tensor of shape \((\text{out\_channels})\) . Default: None stride – the stride of the convolving kernel. Can be a single number or a tuple (sH, sW) . Default: 1 padding – implicit paddings on both sides of the input. Can be a single number or a tuple (padH, padW) . Default: 0 dilation – the spacing between kernel elements. Can be a single number or a tuple (dH, dW) . Default: 1 groups – split input into groups, both \(\text{in\_channels}\) and \(\text{out\_channels}\) should be divisible by the number of groups. Default: 1 Examples: >>> # With square kernels and equal stride >>> filters = tp . randn ( 8 , 4 , 3 , 3 ) >>> inputs = tp . randn ( 1 , 4 , 5 , 5 ) >>> F . conv2d ( inputs , filters , padding = 1 ) # tensorplay.nn.functional.conv3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.conv3d.html tensorplay.nn.functional.conv3d tensorplay.nn.functional. conv3d ( input , weight , bias = None , stride = 1 , padding = 0 , dilation = 1 , groups = 1 ) [source] Applies a 3D convolution over an input image composed of several input planes. See Conv3d for details and output shape. Parameters : input – input tensor of shape \((\text{minibatch} , \text{in\_channels} , iD, iH , iW)\) weight – filters of shape \((\text{out\_channels} , \frac{\text{in\_channels}}{\text{groups}} , kD, kH , kW)\) bias – optional bias tensor of shape \((\text{out\_channels})\) . Default: None stride – the stride of the convolving kernel. Can be a single number or a tuple (sD, sH, sW) . Default: 1 padding – implicit paddings on both sides of the input. Can be a single number or a tuple (padD, padH, padW) . Default: 0 dilation – the spacing between kernel elements. Can be a single number or a tuple (dD, dH, dW) . Default: 1 groups – split input into groups, both \(\text{in\_channels}\) and \(\text{out\_channels}\) should be divisible by the number of groups. Default: 1 Examples: >>> # With square kernels and equal stride >>> filters = tp . randn ( 8 , 4 , 3 , 3 , 3 ) >>> inputs = tp . randn ( 1 , 4 , 5 , 5 , 5 ) >>> F . conv3d ( inputs , filters , padding = 1 ) # tensorplay.nn.functional.cosine_embedding_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.cosine_embedding_loss.html tensorplay.nn.functional.cosine_embedding_loss tensorplay.nn.functional. cosine_embedding_loss ( input1 : TensorBase , input2 : TensorBase , target : TensorBase , margin : float = 0 , size_average = None , reduce = None , reduction : str = 'mean' ) → TensorBase [source] Compute the cosine embedding loss. See CosineEmbeddingLoss for details. # tensorplay.nn.functional.cosine_similarity URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.cosine_similarity.html tensorplay.nn.functional.cosine_similarity tensorplay.nn.functional. cosine_similarity ( x1 : TensorBase , x2 : TensorBase , dim : int = 1 , eps : float = 1e-08 ) → TensorBase [source] Returns cosine similarity between x1 and x2, computed along dim. # tensorplay.nn.functional.cross_entropy URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.cross_entropy.html tensorplay.nn.functional.cross_entropy tensorplay.nn.functional. cross_entropy ( input , target , weight = None , size_average = None , ignore_index = -100 , reduce = None , reduction = 'mean' , label_smoothing = 0.0 ) [source] Compute the cross entropy loss between input logits and target. Port of at::native::cross_entropy_loss_symint (aten/src/ATen/native/LossNLL.cpp:633): equal input/target shapes select the class-probability path, positive label_smoothing blends the NLL with a smoothed uniform term, and otherwise this is nll_loss(log_softmax(input), target) with N-d support. See CrossEntropyLoss for details. # tensorplay.nn.functional.ctc_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.ctc_loss.html tensorplay.nn.functional.ctc_loss tensorplay.nn.functional. ctc_loss ( log_probs : TensorBase , targets : TensorBase , input_lengths , target_lengths , blank : int = 0 , reduction : str = 'mean' , zero_infinity : bool = False ) → TensorBase [source] Compute the Connectionist Temporal Classification loss. Vectorized dynamic-programming port of aten/src/ATen/native/LossCTC.cpp (alpha recurrence over the blank-extended target sequence); autograd flows through log_probs via the composed primitives. Parameters : log_probs – \((T, N, C)\) or \((T, C)\) log-softmax outputs. targets – \((N, S)\) or concatenated \((\sum S_n,)\) . target_lengths ( input_lengths / ) – \((N,)\) or scalars. blank – index of the blank label. Default: 0. reduction – 'none' | 'mean' | 'sum' . zero_infinity – zero out infinite losses (targets too long for T). # tensorplay.nn.functional.dropout URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.dropout.html tensorplay.nn.functional.dropout tensorplay.nn.functional. dropout ( input , p = 0.5 , training = True , inplace = False ) [source] # tensorplay.nn.functional.dropout1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.dropout1d.html tensorplay.nn.functional.dropout1d tensorplay.nn.functional. dropout1d ( input : TensorBase , p : float = 0.5 , training : bool = True , inplace : bool = False ) → TensorBase [source] Randomly zero out entire channels (a channel is a 1D feature map). See Dropout1d for details. # tensorplay.nn.functional.dropout2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.dropout2d.html tensorplay.nn.functional.dropout2d tensorplay.nn.functional. dropout2d ( input , p = 0.5 , training = True , inplace = False ) [source] # tensorplay.nn.functional.dropout3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.dropout3d.html tensorplay.nn.functional.dropout3d tensorplay.nn.functional. dropout3d ( input , p = 0.5 , training = True , inplace = False ) [source] # DType URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.DType.html DType class tensorplay.nn.functional. DType Members: uint8 int8 int16 int32 int64 uint16 uint32 uint64 float16 bfloat16 float32 float64 complex32 complex64 complex128 bcomplex32 bool undefined property name # tensorplay.nn.functional.elu URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.elu.html tensorplay.nn.functional.elu tensorplay.nn.functional. elu ( input , alpha = 1 , inplace = False ) → Tensor [source] # tensorplay.nn.functional.embedding_bag URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.embedding_bag.html tensorplay.nn.functional.embedding_bag tensorplay.nn.functional. embedding_bag ( input : TensorBase , weight : TensorBase , offsets = None , max_norm = None , norm_type : float = 2 , scale_grad_by_freq : bool = False , mode : str = 'mean' , sparse : bool = False , per_sample_weights = None , include_last_offset : bool = False , padding_idx = None ) → TensorBase [source] Compute sums, means or maxes of bags of embeddings. Composition of aten::embedding_bag from dispatched primitives; supports 1-D inputs with offsets (incl. include_last_offset ), fixed length 2-D inputs, per_sample_weights (sum mode), padding_idx exclusion and max_norm renormalization. See tensorplay.nn.EmbeddingBag for details. # tensorplay.nn.functional.embedding URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.embedding.html tensorplay.nn.functional.embedding tensorplay.nn.functional. embedding ( input , weight , padding_idx = None , max_norm = None , norm_type = 2.0 , scale_grad_by_freq = False , sparse = False ) [source] # tensorplay.nn.functional.feature_alpha_dropout URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.feature_alpha_dropout.html tensorplay.nn.functional.feature_alpha_dropout tensorplay.nn.functional. feature_alpha_dropout ( input : TensorBase , p : float = 0.5 , training : bool = False , inplace : bool = False ) → TensorBase [source] Randomly masks out entire channels, setting activations to the negative saturation value of the SELU activation function. See FeatureAlphaDropout for details. # tensorplay.nn.functional.flatten URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.flatten.html tensorplay.nn.functional.flatten tensorplay.nn.functional. flatten ( input , start_dim = 0 , end_dim = -1 ) [source] # tensorplay.nn.functional.fold URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.fold.html tensorplay.nn.functional.fold tensorplay.nn.functional. fold ( input , output_size , kernel_size , dilation = 1 , padding = 0 , stride = 1 ) [source] Combine an array of sliding local blocks into a tensor containing them all (torch F.fold, i.e. aten::col2im). # tensorplay.nn.functional.fractional_max_pool2d_with_indices URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.fractional_max_pool2d_with_indices.html tensorplay.nn.functional.fractional_max_pool2d_with_indices tensorplay.nn.functional. fractional_max_pool2d_with_indices ( input : TensorBase , kernel_size , output_size = None , output_ratio = None , return_indices : bool = True , _random_samples = None ) [source] Applies 2D fractional max pooling over an input signal composed of several input planes, returning (output, indices) . The max-pooling operation is applied in \(kH \times kW\) regions by a stochastic step size determined by the target output size. The number of output features is equal to the number of input planes. Parameters : kernel_size – the size of the window, k or (kH, kW) output_size – target output size oH x oW output_ratio – alternative to output_size, in range (0, 1) return_indices – return pooling indices as well _random_samples – optional (B, C, 2) random starts override See FractionalMaxPool2d for details. # tensorplay.nn.functional.fractional_max_pool2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.fractional_max_pool2d.html tensorplay.nn.functional.fractional_max_pool2d tensorplay.nn.functional. fractional_max_pool2d ( input : TensorBase , kernel_size , output_size = None , output_ratio = None , return_indices : bool = False , _random_samples = None ) [source] Applies 2D fractional max pooling over an input signal. If return_indices is True , returns (output, indices) ; otherwise just the output. # tensorplay.nn.functional.fractional_max_pool3d_with_indices URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.fractional_max_pool3d_with_indices.html tensorplay.nn.functional.fractional_max_pool3d_with_indices tensorplay.nn.functional. fractional_max_pool3d_with_indices ( input : TensorBase , kernel_size , output_size = None , output_ratio = None , return_indices : bool = True , _random_samples = None ) [source] Applies 3D fractional max pooling over an input signal composed of several input planes, returning (output, indices) . Each plane consumes three random samples ordered (T, H, W) , matching aten/src/ATen/native/FractionalMaxPool3d.cpp. # tensorplay.nn.functional.fractional_max_pool3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.fractional_max_pool3d.html tensorplay.nn.functional.fractional_max_pool3d tensorplay.nn.functional. fractional_max_pool3d ( input : TensorBase , kernel_size , output_size = None , output_ratio = None , return_indices : bool = False , _random_samples = None ) [source] Applies 3D fractional max pooling over an input signal. If return_indices is True , returns (output, indices) ; otherwise just the output. # tensorplay.nn.functional.gaussian_nll_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.gaussian_nll_loss.html tensorplay.nn.functional.gaussian_nll_loss tensorplay.nn.functional. gaussian_nll_loss ( input : TensorBase , target : TensorBase , var , full : bool = False , eps : float = 1e-06 , reduction : str = 'mean' ) → TensorBase [source] Compute the Gaussian negative log likelihood loss. See GaussianNLLLoss for details. # tensorplay.nn.functional.gelu URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.gelu.html tensorplay.nn.functional.gelu tensorplay.nn.functional. gelu ( input , approximate = 'none' ) → Tensor [source] When approximate is ‘none’, applies \(\text{GELU}(x) = x * \Phi(x)\) ; ‘tanh’ uses the tanh estimation. # tensorplay.nn.functional.glu URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.glu.html tensorplay.nn.functional.glu tensorplay.nn.functional. glu ( input , dim = -1 ) → Tensor [source] Gated Linear Unit: \(a * \sigma(b)\) where the input is split in half along dim . # tensorplay.nn.functional.grid_sample URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.grid_sample.html tensorplay.nn.functional.grid_sample tensorplay.nn.functional. grid_sample ( input : TensorBase , grid : TensorBase , mode : str = 'bilinear' , padding_mode : str = 'zeros' , align_corners = None ) → TensorBase [source] Compute grid sample. Given an input and a flow-field grid , computes the output using input values and pixel locations from grid . Currently, only spatial (4-D) and volumetric (5-D) input are supported. Parameters : input ( Tensor ) – input of shape \((N, C, H_\text{in}, W_\text{in})\) (4-D case) or \((N, C, D_\text{in}, H_\text{in}, W_\text{in})\) (5-D case) grid ( Tensor ) – flow-field of shape \((N, H_\text{out}, W_\text{out}, 2)\) (4-D case) or \((N, D_\text{out}, H_\text{out}, W_\text{out}, 3)\) (5-D case) mode ( str ) – 'bilinear' | 'nearest' | 'bicubic' . Default: 'bilinear' padding_mode ( str ) – 'zeros' | 'border' | 'reflection' . Default: 'zeros' align_corners ( bool , optional ) – extrema treatment, default False . Composed from dispatched primitives following aten/src/ATen/native/GridSamplerUtils.h; autograd flows to both input and grid . # tensorplay.nn.functional.group_norm URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.group_norm.html tensorplay.nn.functional.group_norm tensorplay.nn.functional. group_norm ( input , num_groups , weight = None , bias = None , eps = 1e-05 ) [source] # tensorplay.nn.functional.grouped_mm URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.grouped_mm.html tensorplay.nn.functional.grouped_mm tensorplay.nn.functional. grouped_mm ( * args , ** kwargs ) [source] # tensorplay.nn.functional.gumbel_softmax URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.gumbel_softmax.html tensorplay.nn.functional.gumbel_softmax tensorplay.nn.functional. gumbel_softmax ( logits : TensorBase , tau : float = 1 , hard : bool = False , eps : float = 1e-10 , dim : int = -1 ) → TensorBase [source] Sample from the Gumbel-Softmax distribution and optionally discretize. Port of torch.nn.functional.gumbel_softmax (reparametrization trick; straight-through when hard=True ). # tensorplay.nn.functional.hardshrink URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.hardshrink.html tensorplay.nn.functional.hardshrink tensorplay.nn.functional. hardshrink ( input , lambd = 0.5 ) → Tensor [source] Applies the hard shrinkage function element-wise. See Hardshrink for more details. # tensorplay.nn.functional.hardsigmoid URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.hardsigmoid.html tensorplay.nn.functional.hardsigmoid tensorplay.nn.functional. hardsigmoid ( input , inplace = False ) → Tensor [source] # tensorplay.nn.functional.hardswish URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.hardswish.html tensorplay.nn.functional.hardswish tensorplay.nn.functional. hardswish ( input , inplace = False ) → Tensor [source] # tensorplay.nn.functional.hardtanh URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.hardtanh.html tensorplay.nn.functional.hardtanh tensorplay.nn.functional. hardtanh ( input , min_val = -1.0 , max_val = 1.0 , inplace = False ) → Tensor [source] # tensorplay.nn.functional.hinge_embedding_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.hinge_embedding_loss.html tensorplay.nn.functional.hinge_embedding_loss tensorplay.nn.functional. hinge_embedding_loss ( input : TensorBase , target : TensorBase , margin : float = 1.0 , size_average = None , reduce = None , reduction : str = 'mean' ) → TensorBase [source] Compute the hinge embedding loss. See HingeEmbeddingLoss for details. # tensorplay.nn.functional.huber_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.huber_loss.html tensorplay.nn.functional.huber_loss tensorplay.nn.functional. huber_loss ( input : TensorBase , target : TensorBase , reduction : str = 'mean' , delta : float = 1.0 , weight : TensorBase | None = None ) → TensorBase [source] Compute the Huber loss, with optional weighting. Function uses a squared term if the absolute error falls below delta and a delta-scaled L1 term otherwise. See HuberLoss for details. # tensorplay.nn.functional.instance_norm URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.instance_norm.html tensorplay.nn.functional.instance_norm tensorplay.nn.functional. instance_norm ( input , running_mean = None , running_var = None , weight = None , bias = None , use_input_stats = True , momentum = 0.1 , eps = 1e-05 ) [source] # tensorplay.nn.functional.interpolate URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.interpolate.html tensorplay.nn.functional.interpolate tensorplay.nn.functional. interpolate ( input : TensorBase , size = None , scale_factor = None , mode : str = 'nearest' , align_corners = None , recompute_scale_factor = None , antialias : bool = False ) → TensorBase [source] interpolate(input, size=None, scale_factor=None, mode=’nearest’, align_corners=None) -> Tensor Routes to the native upsample_* ops exactly like torch.nn.functional.interpolate routes to ATen’s upsample family. # tensorplay.nn.functional.kl_div URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.kl_div.html tensorplay.nn.functional.kl_div tensorplay.nn.functional. kl_div ( input : TensorBase , target : TensorBase , size_average = None , reduce = None , reduction : str = 'mean' , log_target : bool = False ) → TensorBase [source] Compute the KL Divergence loss. input holds log-probabilities; see KLDivLoss . Note that reduction='mean' divides by the number of elements and does not return the true KL divergence value — use 'batchmean' . # tensorplay.nn.functional.l1_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.l1_loss.html tensorplay.nn.functional.l1_loss tensorplay.nn.functional. l1_loss ( input : TensorBase , target : TensorBase , size_average = None , reduce = None , reduction : str = 'mean' , weight : TensorBase | None = None ) → TensorBase [source] Compute the L1 loss, with optional weighting. Function that takes the mean element-wise absolute value difference. See L1Loss for details. # tensorplay.nn.functional.layer_norm URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.layer_norm.html tensorplay.nn.functional.layer_norm tensorplay.nn.functional. layer_norm ( input , normalized_shape , weight = None , bias = None , eps = 1e-05 ) [source] # tensorplay.nn.functional.leaky_relu URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.leaky_relu.html tensorplay.nn.functional.leaky_relu tensorplay.nn.functional. leaky_relu ( input , negative_slope = 0.01 , inplace = False ) → Tensor [source] # tensorplay.nn.functional.linear_cross_entropy URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.linear_cross_entropy.html tensorplay.nn.functional.linear_cross_entropy tensorplay.nn.functional. linear_cross_entropy ( input : TensorBase , linear_weight : TensorBase , target : TensorBase , * , linear_bias = None , weight = None , reduction : str = 'mean' , ignore_index = None , label_smoothing : float = 0.0 , options = None ) → TensorBase [source] Compute cross entropy between input , transformed linearly, and target. Equivalent to cross_entropy(linear(input, linear_weight), target, **kwargs) (reference path; chunked/fused options are ignored). # tensorplay.nn.functional.linear URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.linear.html tensorplay.nn.functional.linear tensorplay.nn.functional. linear ( input : TensorBase , weight : TensorBase , bias : TensorBase | None = None ) → TensorBase [source] Applies a linear transformation to the incoming data: \(y = xA^T + b\) . Shape: Input: \((*, H_\text{in})\) where \(*\) means any number of dimensions including none and \(H_\text{in} = \text{in\_features}\) . Weight: \((H_\text{out}, H_\text{in})\) where \(H_\text{out} = \text{out\_features}\) . Bias: \((H_\text{out})\) Output: \((*, H_\text{out})\) See Linear for more details. # tensorplay.nn.functional.local_response_norm URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.local_response_norm.html tensorplay.nn.functional.local_response_norm tensorplay.nn.functional. local_response_norm ( input : TensorBase , size : int , alpha : float = 0.0001 , beta : float = 0.75 , k : float = 1.0 ) → TensorBase [source] Apply local response normalization over an input signal. The input signal is composed of several input planes, where channels occupy the second dimension. Normalization is applied across channels. See LocalResponseNorm for details. # tensorplay.nn.functional.log_softmax URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.log_softmax.html tensorplay.nn.functional.log_softmax tensorplay.nn.functional. log_softmax ( input , dim = None , dtype = None ) [source] # tensorplay.nn.functional.logsigmoid URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.logsigmoid.html tensorplay.nn.functional.logsigmoid tensorplay.nn.functional. logsigmoid ( input ) → Tensor [source] Applies element-wise \(\text{LogSigmoid}(x_i) = \log \left(\frac{1}{1 + \exp(-x_i)}\right)\) See LogSigmoid for more details. # tensorplay.nn.functional.lp_pool1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.lp_pool1d.html tensorplay.nn.functional.lp_pool1d tensorplay.nn.functional. lp_pool1d ( input : TensorBase , norm_type , kernel_size , stride = None , ceil_mode : bool = False ) → TensorBase [source] Apply a 1D power-average pooling over an input signal. See LPPool1d for details. # tensorplay.nn.functional.lp_pool2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.lp_pool2d.html tensorplay.nn.functional.lp_pool2d tensorplay.nn.functional. lp_pool2d ( input : TensorBase , norm_type , kernel_size , stride = None , ceil_mode : bool = False ) → TensorBase [source] Apply a 2D power-average pooling over an input signal. If the sum of all inputs to the power of p is zero, the gradient is set to zero as well. See LPPool2d for details. # tensorplay.nn.functional.lp_pool3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.lp_pool3d.html tensorplay.nn.functional.lp_pool3d tensorplay.nn.functional. lp_pool3d ( input : TensorBase , norm_type , kernel_size , stride = None , ceil_mode : bool = False ) → TensorBase [source] Apply a 3D power-average pooling over an input signal. See LPPool3d for details. # tensorplay.nn.functional.margin_ranking_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.margin_ranking_loss.html tensorplay.nn.functional.margin_ranking_loss tensorplay.nn.functional. margin_ranking_loss ( input1 : TensorBase , input2 : TensorBase , target : TensorBase , margin : float = 0 , size_average = None , reduce = None , reduction : str = 'mean' ) → TensorBase [source] Compute the margin ranking loss. See MarginRankingLoss for details. # tensorplay.nn.functional.max_pool1d_with_indices URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.max_pool1d_with_indices.html tensorplay.nn.functional.max_pool1d_with_indices tensorplay.nn.functional. max_pool1d_with_indices ( input : TensorBase , kernel_size , stride = None , padding = 0 , dilation = 1 , ceil_mode : bool = False , return_indices : bool = True ) [source] Applies a 1D max pooling over an input signal, returning (output, indices) . See MaxPool1d for details. # tensorplay.nn.functional.max_pool1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.max_pool1d.html tensorplay.nn.functional.max_pool1d tensorplay.nn.functional. max_pool1d ( input : TensorBase , kernel_size , stride = None , padding = 0 , dilation = 1 , ceil_mode : bool = False , return_indices : bool = False ) [source] max_pool1d(input, kernel_size, stride=None, padding=0, dilation=1, ceil_mode=False, return_indices=False) -> Tensor Applies a 1D max pooling over an input signal composed of several input planes. Input shape (N, C, L) or unbatched (C, L) . # tensorplay.nn.functional.max_pool2d_with_indices URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.max_pool2d_with_indices.html tensorplay.nn.functional.max_pool2d_with_indices tensorplay.nn.functional. max_pool2d_with_indices ( input : TensorBase , kernel_size , stride = None , padding = 0 , dilation = 1 , ceil_mode : bool = False , return_indices : bool = True ) [source] Applies a 2D max pooling over an input composed of several input planes, returning (output, indices) . See MaxPool2d for details. # tensorplay.nn.functional.max_pool2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.max_pool2d.html tensorplay.nn.functional.max_pool2d tensorplay.nn.functional. max_pool2d ( input , kernel_size , stride = None , padding = 0 , dilation = 1 , ceil_mode = False , return_indices = False ) [source] # tensorplay.nn.functional.max_pool3d_with_indices URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.max_pool3d_with_indices.html tensorplay.nn.functional.max_pool3d_with_indices tensorplay.nn.functional. max_pool3d_with_indices ( input : TensorBase , kernel_size , stride = None , padding = 0 , dilation = 1 , ceil_mode : bool = False , return_indices : bool = True ) [source] Applies a 3D max pooling over an input signal, returning (output, indices) . See MaxPool3d for details. # tensorplay.nn.functional.max_pool3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.max_pool3d.html tensorplay.nn.functional.max_pool3d tensorplay.nn.functional. max_pool3d ( input : TensorBase , kernel_size , stride = None , padding = 0 , dilation = 1 , ceil_mode : bool = False , return_indices : bool = False ) → TensorBase [source] max_pool3d(input, kernel_size, stride=None, padding=0, dilation=1, ceil_mode=False, return_indices=False) -> Tensor Applies a 3D max pooling over an input signal composed of several input planes. Input shape (N, C, D, H, W) or unbatched (C, D, H, W) . # tensorplay.nn.functional.max_unpool1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.max_unpool1d.html tensorplay.nn.functional.max_unpool1d tensorplay.nn.functional. max_unpool1d ( input : TensorBase , indices : TensorBase , kernel_size , stride = None , padding = 0 , output_size = None ) → TensorBase [source] Compute a partial inverse of MaxPool1d . See MaxUnpool1d for details. # tensorplay.nn.functional.max_unpool2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.max_unpool2d.html tensorplay.nn.functional.max_unpool2d tensorplay.nn.functional. max_unpool2d ( input : TensorBase , indices : TensorBase , kernel_size , stride = None , padding = 0 , output_size = None ) → TensorBase [source] Compute a partial inverse of MaxPool2d . See MaxUnpool2d for details. # tensorplay.nn.functional.max_unpool3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.max_unpool3d.html tensorplay.nn.functional.max_unpool3d tensorplay.nn.functional. max_unpool3d ( input : TensorBase , indices : TensorBase , kernel_size , stride = None , padding = 0 , output_size = None ) → TensorBase [source] Compute a partial inverse of MaxPool3d . See MaxUnpool3d for details. # tensorplay.nn.functional.mish URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.mish.html tensorplay.nn.functional.mish tensorplay.nn.functional. mish ( input , inplace = False ) → Tensor [source] # tensorplay.nn.functional.mse_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.mse_loss.html tensorplay.nn.functional.mse_loss tensorplay.nn.functional. mse_loss ( input , target , reduction = 'mean' ) [source] # tensorplay.nn.functional.multi_head_attention_forward URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.multi_head_attention_forward.html tensorplay.nn.functional.multi_head_attention_forward tensorplay.nn.functional. multi_head_attention_forward ( query : TensorBase , key : TensorBase , value : TensorBase , embed_dim_to_check : int , num_heads : int , in_proj_weight : TensorBase | None = None , in_proj_bias : TensorBase | None = None , bias_k = None , bias_v = None , add_zero_attn : bool = False , dropout_p : float = 0.0 , out_proj_weight : TensorBase | None = None , out_proj_bias : TensorBase | None = None , training : bool = True , key_padding_mask = None , need_weights : bool = True , attn_mask = None , use_separate_proj_weight : bool = False , q_proj_weight = None , k_proj_weight = None , v_proj_weight = None , static_k = None , static_v = None , average_attn_weights : bool = True , is_causal : bool = False ) [source] torch-compatible multi_head_attention_forward. Follows the structure of third_party/pytorch/aten/src/ATen/native/transformers/attention.cpp _scaled_dot_product_attention paths composed from dispatched primitives. # tensorplay.nn.functional.multi_margin_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.multi_margin_loss.html tensorplay.nn.functional.multi_margin_loss tensorplay.nn.functional. multi_margin_loss ( input : TensorBase , target : TensorBase , p : int = 1 , margin : float = 1.0 , weight : TensorBase | None = None , size_average = None , reduce = None , reduction : str = 'mean' ) → TensorBase [source] Compute the multi margin loss, with optional weighting. Vectorized composition of aten/src/ATen/native/LossMultiMargin.cpp: sum_d max(0, margin - x_y + x_d)^p * w_y / C over non-target classes. See MultiMarginLoss for details. # tensorplay.nn.functional.multilabel_margin_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.multilabel_margin_loss.html tensorplay.nn.functional.multilabel_margin_loss tensorplay.nn.functional. multilabel_margin_loss ( input : TensorBase , target : TensorBase , size_average = None , reduce = None , reduction : str = 'mean' ) → TensorBase [source] Compute the multilabel margin loss. Composition of aten/src/ATen/native/LossMultiLabelMargin.cpp: for each positive label y (targets are active until the first -1 ), add max(0, 1 - x[y] + x[d]) over non-target labels d ; divide by C. See MultiLabelMarginLoss for details. # tensorplay.nn.functional.multilabel_soft_margin_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.multilabel_soft_margin_loss.html tensorplay.nn.functional.multilabel_soft_margin_loss tensorplay.nn.functional. multilabel_soft_margin_loss ( input : TensorBase , target : TensorBase , weight : TensorBase | None = None , size_average = None , reduce = None , reduction : str = 'mean' ) → TensorBase [source] Compute the multilabel soft margin loss. See MultiLabelSoftMarginLoss for details. # tensorplay.nn.functional.native_channel_shuffle URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.native_channel_shuffle.html tensorplay.nn.functional.native_channel_shuffle tensorplay.nn.functional. native_channel_shuffle ( input : TensorBase , groups : int ) → TensorBase [source] Native channel shuffle primitive (torch.native_channel_shuffle). # tensorplay.nn.functional.nll_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.nll_loss.html tensorplay.nn.functional.nll_loss tensorplay.nn.functional. nll_loss ( input , target , weight = None , size_average = None , ignore_index = -100 , reduce = None , reduction = 'mean' ) [source] The negative log likelihood loss. Supports the same shapes as torch.nn.functional.nll_loss() : 1D input with a scalar target , 2D (N, C) , and N-d (N, C, d_1, ..., d_k) (torch’s nll_loss_nd ) with matching target . See NLLLoss for details. # tensorplay.nn.functional.normalize URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.normalize.html tensorplay.nn.functional.normalize tensorplay.nn.functional. normalize ( input , p = 2 , dim = 1 , eps = 1e-12 ) → Tensor [source] Performs \(L_p\) normalization over the specified dimension — torch.nn.functional.normalize divides by clamp_min(norm, eps). # tensorplay.nn.functional.one_hot URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.one_hot.html tensorplay.nn.functional.one_hot tensorplay.nn.functional. one_hot ( tensor , num_classes = -1 ) → LongTensor [source] Returns long tensor shaped tensor.shape + (num_classes,) with a 1 at each label position — port of ATen one_hot. # tensorplay.nn.functional.pad URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.pad.html tensorplay.nn.functional.pad tensorplay.nn.functional. pad ( input , pad , mode = 'constant' , value = 0 ) [source] Pads tensor. pad values are described starting from the last dimension and moving forward, exactly like torch.nn.functional.pad. Non-constant modes support the last 3 dimensions of a 3D/4D/5D input (torch’s restriction). # tensorplay.nn.functional.pairwise_distance URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.pairwise_distance.html tensorplay.nn.functional.pairwise_distance tensorplay.nn.functional. pairwise_distance ( x1 : TensorBase , x2 : TensorBase , p : float = 2.0 , eps : float = 1e-06 , keepdim : bool = False ) → TensorBase [source] Computes the pairwise distance between input vectors. Port of aten::pairwise_distance: norm(x1 - x2 + eps, p) over the last dimension. # tensorplay.nn.functional.pdist URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.pdist.html tensorplay.nn.functional.pdist tensorplay.nn.functional. pdist ( input : TensorBase , p : float = 2.0 ) → TensorBase [source] Computes the pairwise distance between rows of input . Returns the flattened upper triangle of the N x N distance matrix — port of aten::pdist semantics. # tensorplay.nn.functional.pixel_shuffle URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.pixel_shuffle.html tensorplay.nn.functional.pixel_shuffle tensorplay.nn.functional. pixel_shuffle ( input : TensorBase , upscale_factor : int ) → TensorBase [source] Rearranges elements in a tensor of shape (*, C x r^2, H, W) to a tensor of shape (*, C, H x r, W x r) . Port of aten::pixel_shuffle: output[n, c, h*r+i, w*r+j] = input[n, c*r^2 + i*r + j, h, w] . # tensorplay.nn.functional.pixel_unshuffle URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.pixel_unshuffle.html tensorplay.nn.functional.pixel_unshuffle tensorplay.nn.functional. pixel_unshuffle ( input : TensorBase , downscale_factor : int ) → TensorBase [source] Reverses the pixel_shuffle() transformation: (*, C, H x r, W x r) -> (*, C x r^2, H, W) . # tensorplay.nn.functional.poisson_nll_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.poisson_nll_loss.html tensorplay.nn.functional.poisson_nll_loss tensorplay.nn.functional. poisson_nll_loss ( input : TensorBase , target : TensorBase , log_input : bool = True , full : bool = False , size_average = None , eps : float = 1e-08 , reduce = None , reduction : str = 'mean' ) → TensorBase [source] Compute the Poisson negative log likelihood loss. See PoissonNLLLoss for details. # tensorplay.nn.functional.prelu URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.prelu.html tensorplay.nn.functional.prelu tensorplay.nn.functional. prelu ( input , weight ) [source] # tensorplay.nn.functional.relu URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.relu.html tensorplay.nn.functional.relu tensorplay.nn.functional. relu ( input , inplace = False ) [source] # tensorplay.nn.functional.relu6 URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.relu6.html tensorplay.nn.functional.relu6 tensorplay.nn.functional. relu6 ( input , inplace = False ) → Tensor [source] ReLU6: \(\min(\max(0, x), 6)\) — torch.nn.functional.relu6. # tensorplay.nn.functional.rms_norm URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.rms_norm.html tensorplay.nn.functional.rms_norm tensorplay.nn.functional. rms_norm ( input : TensorBase , normalized_shape , weight : TensorBase | None = None , eps : float | None = None ) → TensorBase [source] Apply Root Mean Square Layer Normalization — composed per the ATen rms_norm composite (fp32 compute for reduced dtypes). # tensorplay.nn.functional.rrelu URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.rrelu.html tensorplay.nn.functional.rrelu tensorplay.nn.functional. rrelu ( input , lower = 1. / 8 , upper = 1. / 3 , training = False , inplace = False ) → Tensor [source] Randomized leaky ReLU. See RReLU for more details. # tensorplay.nn.functional.scaled_dot_product_attention URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.scaled_dot_product_attention.html tensorplay.nn.functional.scaled_dot_product_attention tensorplay.nn.functional. scaled_dot_product_attention ( query : TensorBase , key : TensorBase , value : TensorBase , attn_mask = None , dropout_p : float = 0.0 , is_causal : bool = False , scale = None , backend : str | None = None ) → TensorBase [source] scaled_dot_product_attention(query, key, value, attn_mask=None, dropout_p=0.0, is_causal=False, scale=None, backend=None) -> Tensor Computes scaled dot product attention on query, key and value. Routes to the fused native kernel when possible, otherwise follows torch’s math reference: \[\text{Attention}(Q, K, V) = \text{softmax}(\frac{Q K^T}{\sqrt{E}}) V\] Parameters : backend ( str , optional ) – 'flash' | 'mem_efficient' | 'math' , or None to pick automatically. 'flash' selects the fused flash-attention kernel (impl=1), 'math' forces the composed reference path; 'mem_efficient' , matching torch’s memory-efficient backend, is not available in this build. # tensorplay.nn.functional.scaled_grouped_mm URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.scaled_grouped_mm.html tensorplay.nn.functional.scaled_grouped_mm tensorplay.nn.functional. scaled_grouped_mm ( * args , ** kwargs ) [source] # tensorplay.nn.functional.scaled_mm URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.scaled_mm.html tensorplay.nn.functional.scaled_mm tensorplay.nn.functional. scaled_mm ( * args , ** kwargs ) [source] # tensorplay.nn.functional.selu URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.selu.html tensorplay.nn.functional.selu tensorplay.nn.functional. selu ( input , inplace = False ) → Tensor [source] # tensorplay.nn.functional.sigmoid URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.sigmoid.html tensorplay.nn.functional.sigmoid tensorplay.nn.functional. sigmoid ( input ) → Tensor [source] Applies the element-wise function \(\text{Sigmoid}(x) = \frac{1}{1 + \exp(-x)}\) # tensorplay.nn.functional.silu URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.silu.html tensorplay.nn.functional.silu tensorplay.nn.functional. silu ( input : TensorBase , inplace : bool = False ) → TensorBase [source] Apply the Sigmoid Linear Unit (SiLU) function, element-wise. The SiLU function is also known as the swish function. \[\text{silu}(x) = x * \sigma(x), \text{where } \sigma(x) \text{ is the logistic sigmoid.}\] Note See Gaussian Error Linear Units (GELUs) where the SiLU (Sigmoid Linear Unit) was originally coined, and see Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning and Swish: a Self-Gated Activation Function where the SiLU was experimented with later. See SiLU for more details. # tensorplay.nn.functional.smooth_l1_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.smooth_l1_loss.html tensorplay.nn.functional.smooth_l1_loss tensorplay.nn.functional. smooth_l1_loss ( input : TensorBase , target : TensorBase , size_average = None , reduce = None , reduction : str = 'mean' , beta : float = 1.0 ) → TensorBase [source] Compute the Smooth L1 loss. Function uses a squared term if the absolute element-wise error falls below beta and an L1 term otherwise. See SmoothL1Loss for details. # tensorplay.nn.functional.soft_margin_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.soft_margin_loss.html tensorplay.nn.functional.soft_margin_loss tensorplay.nn.functional. soft_margin_loss ( input : TensorBase , target : TensorBase , size_average = None , reduce = None , reduction : str = 'mean' ) → TensorBase [source] Compute the soft margin loss. See SoftMarginLoss for details. # tensorplay.nn.functional.softmax URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.softmax.html tensorplay.nn.functional.softmax tensorplay.nn.functional. softmax ( input , dim = None , dtype = None ) [source] # tensorplay.nn.functional.softmin URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.softmin.html tensorplay.nn.functional.softmin tensorplay.nn.functional. softmin ( input : TensorBase , dim : int | None = None , dtype = None ) → TensorBase [source] Apply a softmin function. Note that \(\text{Softmin}(x) = \text{Softmax}(-x)\) . See Softmin for more details. # tensorplay.nn.functional.softplus URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.softplus.html tensorplay.nn.functional.softplus tensorplay.nn.functional. softplus ( input , beta = 1 , threshold = 20 ) → Tensor [source] # tensorplay.nn.functional.softshrink URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.softshrink.html tensorplay.nn.functional.softshrink tensorplay.nn.functional. softshrink ( input , lambd = 0.5 ) → Tensor [source] Applies the soft shrinkage function element-wise. See Softshrink for more details. # tensorplay.nn.functional.softsign URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.softsign.html tensorplay.nn.functional.softsign tensorplay.nn.functional. softsign ( input ) → Tensor [source] Applies element-wise, the function \(\text{SoftSign}(x) = \frac{x}{1 + |x|}\) See Softsign for more details. # tensorplay.nn.functional.tanh URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.tanh.html tensorplay.nn.functional.tanh tensorplay.nn.functional. tanh ( input ) → Tensor [source] Applies element-wise \(\text{Tanh}(x) = \frac{\exp(x) - \exp(-x)}{\exp(x) + \exp(-x)}\) # tensorplay.nn.functional.tanhshrink URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.tanhshrink.html tensorplay.nn.functional.tanhshrink tensorplay.nn.functional. tanhshrink ( input ) → Tensor [source] Applies element-wise, \(\text{Tanhshrink}(x) = x - \text{Tanh}(x)\) See Tanhshrink for more details. # Tensor URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.Tensor.html Tensor tensorplay.nn.functional. Tensor alias of TensorBase # tensorplay.nn.functional.threshold URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.threshold.html tensorplay.nn.functional.threshold tensorplay.nn.functional. threshold ( input : TensorBase , threshold : float , value : float , inplace : bool = False ) → TensorBase [source] Apply a threshold to each element of the input Tensor. See Threshold for more details. # tensorplay.nn.functional.triplet_margin_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.triplet_margin_loss.html tensorplay.nn.functional.triplet_margin_loss tensorplay.nn.functional. triplet_margin_loss ( anchor : TensorBase , positive : TensorBase , negative : TensorBase , margin : float = 1.0 , p : float = 2 , eps : float = 1e-06 , swap : bool = False , size_average = None , reduce = None , reduction : str = 'mean' ) → TensorBase [source] Compute the triplet loss between given input tensors and a margin greater than 0. See TripletMarginLoss for details. # tensorplay.nn.functional.triplet_margin_with_distance_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.triplet_margin_with_distance_loss.html tensorplay.nn.functional.triplet_margin_with_distance_loss tensorplay.nn.functional. triplet_margin_with_distance_loss ( anchor : TensorBase , positive : TensorBase , negative : TensorBase , * , distance_function = None , margin : float = 1.0 , swap : bool = False , reduction : str = 'mean' ) → TensorBase [source] Compute the triplet margin loss using a custom distance function. See TripletMarginWithDistanceLoss for details. # tensorplay.nn.functional.unfold URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.unfold.html tensorplay.nn.functional.unfold tensorplay.nn.functional. unfold ( input , kernel_size , dilation = 1 , padding = 0 , stride = 1 ) [source] Extract sliding local blocks from a batched input tensor (torch F.unfold, i.e. aten::im2col). 4-D batched input is the common case; unbatched 3-D input is accepted like aten’s im2col. # tensorplay.nn.functional.upsample_bilinear URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.upsample_bilinear.html tensorplay.nn.functional.upsample_bilinear tensorplay.nn.functional. upsample_bilinear ( input , size = None , scale_factor = None ) [source] Upsamples the input using bilinear upsampling. Warning This function is deprecated in favor of interpolate() with mode='bilinear', align_corners=True . # tensorplay.nn.functional.upsample_nearest URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.upsample_nearest.html tensorplay.nn.functional.upsample_nearest tensorplay.nn.functional. upsample_nearest ( input , size = None , scale_factor = None ) [source] Upsamples the input using nearest neighbours. Warning This function is deprecated in favor of interpolate() with mode='nearest' . # tensorplay.nn.functional.upsample URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.functional.upsample.html tensorplay.nn.functional.upsample tensorplay.nn.functional. upsample ( input , size = None , scale_factor = None , mode = 'nearest' , align_corners = None ) [source] Upsamples the input to the given size or scale_factor . Warning This function is deprecated in favor of interpolate() . Equivalent to interpolate(...) . # tensorplay.nn.init.calculate_gain URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.init.calculate_gain.html tensorplay.nn.init.calculate_gain tensorplay.nn.init. calculate_gain ( nonlinearity , param = None ) [source] # tensorplay.nn.init.constant_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.init.constant_.html tensorplay.nn.init.constant_ tensorplay.nn.init. constant_ ( tensor , val ) [source] # tensorplay.nn.init.dirac_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.init.dirac_.html tensorplay.nn.init.dirac_ tensorplay.nn.init. dirac_ ( tensor , groups = 1 ) [source] # tensorplay.nn.init.eye_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.init.eye_.html tensorplay.nn.init.eye_ tensorplay.nn.init. eye_ ( tensor ) [source] # tensorplay.nn.init.kaiming_normal_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.init.kaiming_normal_.html tensorplay.nn.init.kaiming_normal_ tensorplay.nn.init. kaiming_normal_ ( tensor , a = 0 , mode = 'fan_in' , nonlinearity = 'leaky_relu' ) [source] # tensorplay.nn.init.kaiming_uniform_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.init.kaiming_uniform_.html tensorplay.nn.init.kaiming_uniform_ tensorplay.nn.init. kaiming_uniform_ ( tensor , a = 0 , mode = 'fan_in' , nonlinearity = 'leaky_relu' ) [source] # tensorplay.nn.init.normal_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.init.normal_.html tensorplay.nn.init.normal_ tensorplay.nn.init. normal_ ( tensor , mean = 0.0 , std = 1.0 ) [source] # tensorplay.nn.init.ones_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.init.ones_.html tensorplay.nn.init.ones_ tensorplay.nn.init. ones_ ( tensor ) [source] # tensorplay.nn.init.orthogonal_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.init.orthogonal_.html tensorplay.nn.init.orthogonal_ tensorplay.nn.init. orthogonal_ ( tensor , gain = 1 , generator = None ) [source] # tensorplay.nn.init.sparse_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.init.sparse_.html tensorplay.nn.init.sparse_ tensorplay.nn.init. sparse_ ( tensor , sparsity , std = 0.01 , generator = None ) [source] # tensorplay.nn.init.trunc_normal_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.init.trunc_normal_.html tensorplay.nn.init.trunc_normal_ tensorplay.nn.init. trunc_normal_ ( tensor , mean = 0.0 , std = 1.0 , a = -2.0 , b = 2.0 ) [source] Fills the input Tensor with values drawn from a truncated normal distribution. Method is based on the rejection-sampling scheme in https://people.sc.fsu.edu/~jburkardt/presentations/truncated_normal.pdf — a direct port of torch.nn.init._no_grad_trunc_normal_ (torch/nn/init.py), which torchvision transformer models use for positional embeddings. # tensorplay.nn.init.uniform_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.init.uniform_.html tensorplay.nn.init.uniform_ tensorplay.nn.init. uniform_ ( tensor , a = 0.0 , b = 1.0 ) [source] # tensorplay.nn.init.xavier_normal_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.init.xavier_normal_.html tensorplay.nn.init.xavier_normal_ tensorplay.nn.init. xavier_normal_ ( tensor , gain = 1.0 ) [source] # tensorplay.nn.init.xavier_uniform_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.init.xavier_uniform_.html tensorplay.nn.init.xavier_uniform_ tensorplay.nn.init. xavier_uniform_ ( tensor , gain = 1.0 ) [source] # tensorplay.nn.init.zeros_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.init.zeros_.html tensorplay.nn.init.zeros_ tensorplay.nn.init. zeros_ ( tensor ) [source] # CELU URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.CELU.html CELU class tensorplay.nn.modules.activation. CELU ( alpha : float = 1.0 , inplace : bool = False ) [source] Applies celu: max(0, x) + min(0, alpha * (exp(x / alpha) - 1)) . add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # ELU URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.ELU.html ELU class tensorplay.nn.modules.activation. ELU ( alpha : float = 1.0 , inplace : bool = False ) [source] Applies elu: max(0, x) + min(0, alpha * (exp(x) - 1)) . Mirrors torch.nn.ELU. add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # GLU URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.GLU.html GLU class tensorplay.nn.modules.activation. GLU ( dim : int = -1 ) [source] Applies the Gaussian Error Linear Units function. \[\text{GLU}(a, b) = a \otimes \sigma(b)\] where \(a\) is the first half of the input matrices and \(b\) is the second half. Parameters : dim ( int ) – the dimension on which to split the input. Default: -1 add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Hardshrink URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.Hardshrink.html Hardshrink class tensorplay.nn.modules.activation. Hardshrink ( lambd : float = 0.5 ) [source] Applies the Hard Shrinkage (Hardshrink) function element-wise. \[\begin{split}\text{HardShrink}(x) = \begin{cases} x & \text{ if } x > \lambda \\ x & \text{ if } x < -\lambda \\ 0 & \text{ otherwise } \end{cases}\end{split}\] Parameters : lambd – the \(\lambda\) value for the Hardshrink formulation. Default: 0.5 add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Hardsigmoid URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.Hardsigmoid.html Hardsigmoid class tensorplay.nn.modules.activation. Hardsigmoid ( inplace : bool = False ) [source] Applies hardsigmoid, element-wise: ReLU6(x + 3) / 6 . Mirrors torch.nn.Hardsigmoid (ATen Activation.cpp hardsigmoid_kernel). add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Hardswish URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.Hardswish.html Hardswish class tensorplay.nn.modules.activation. Hardswish ( inplace : bool = False ) [source] Applies hardswish, element-wise: x * ReLU6(x + 3) / 6 . Mirrors torch.nn.Hardswish (ATen Activation.cpp hardswish_kernel). add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Hardtanh URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.Hardtanh.html Hardtanh class tensorplay.nn.modules.activation. Hardtanh ( min_val : float = -1.0 , max_val : float = 1.0 , inplace : bool = False , min_value : float | None = None , max_value : float | None = None ) [source] Applies the HardTanh function element-wise. \[\begin{split}\text{HardTanh}(x) = \begin{cases} \text{max\_val} & \text{ if } x > \text{ max\_val } \\ \text{min\_val} & \text{ if } x < \text{ min\_val } \\ x & \text{ otherwise } \\ \end{cases}\end{split}\] Parameters : min_val – minimum value of the linear region range. Default: -1 max_val – maximum value of the linear region range. Default: 1 inplace – can optionally do the operation in-place. Default: False add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # LeakyReLU URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.LeakyReLU.html LeakyReLU class tensorplay.nn.modules.activation. LeakyReLU ( negative_slope : float = 0.01 , inplace : bool = False ) [source] Applies leaky_relu: max(0, x) + negative_slope * min(0, x) . Mirrors torch.nn.LeakyReLU. add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # LogSigmoid URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.LogSigmoid.html LogSigmoid class tensorplay.nn.modules.activation. LogSigmoid ( * args , ** kwargs ) [source] Applies the Logsigmoid function element-wise. \[\text{LogSigmoid}(x) = \log\left(\frac{ 1 }{ 1 + \exp(-x)}\right)\] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Run forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # LogSoftmax URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.LogSoftmax.html LogSoftmax class tensorplay.nn.modules.activation. LogSoftmax ( dim = None ) [source] Log-softmax over dim , mirroring torch.nn.LogSoftmax. add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Mish URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.Mish.html Mish class tensorplay.nn.modules.activation. Mish ( inplace : bool = False ) [source] Applies mish: x * tanh(softplus(x)) . Mirrors torch.nn.Mish (ATen ActivationMishKernel). add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # PReLU URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.PReLU.html PReLU class tensorplay.nn.modules.activation. PReLU ( num_parameters : int = 1 , init : float = 0.25 , device = None , dtype = None ) [source] Applies the element-wise PReLU function. \[\text{PReLU}(x) = \max(0,x) + a * \min(0,x)\] or \[\begin{split}\text{PReLU}(x) = \begin{cases} x, & \text{ if } x \ge 0 \\ ax, & \text{ otherwise } \end{cases}\end{split}\] Here \(a\) is a learnable parameter. When called without arguments, nn.PReLU() uses a single parameter \(a\) across all input channels. If called with nn.PReLU(nChannels) , a separate \(a\) is used for each input channel. Note weight decay should not be used when learning \(a\) for good performance. Note Channel dim is the 2nd dim of input. When input has dims < 2, then there is no channel dim and the number of channels = 1. Parameters : num_parameters ( int ) – number of \(a\) to learn. Although it takes an int as input, there is only two values are legitimate: 1, or the number of channels at input. Default: 1 init ( float ) – the initial value of \(a\) . Default: 0.25 Shape: Input: \(( *)\) where * means, any number of additional dimensions. Output: \((*)\) , same shape as the input. weight the learnable weights of shape ( num_parameters ). Type : Tensor Examples: >>> m = nn . PReLU () >>> input = tensorplay . randn ( 2 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str [source] Return the extra representation of the module. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module reset_parameters ( ) → None [source] Resets parameters based on their initialization used in __init__ . set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # ReLU URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.ReLU.html ReLU class tensorplay.nn.modules.activation. ReLU ( inplace : bool = False ) [source] Applies the rectified linear unit function element-wise. \(\text{ReLU}(x) = (x)^+ = \max(0, x)\) Parameters : inplace – can optionally do the operation in-place. Default: False Shape: Input: \((*)\) , where \(*\) means any number of dimensions. Output: \((*)\) , same shape as the input. Examples: >>> m = nn . ReLU () >>> input = tensorplay . randn ( 2 ) >>> output = m ( input ) An implementation of CReLU - https : // arxiv . org / abs / 1603.05201 >>> m = nn . ReLU () >>> input = tensorplay . randn ( 2 ) . unsqueeze ( 0 ) >>> output = tensorplay . cat (( m ( input ), m ( - input ))) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str [source] Return the extra representation of the module. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # ReLU6 URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.ReLU6.html ReLU6 class tensorplay.nn.modules.activation. ReLU6 ( inplace : bool = False ) [source] Applies the element-wise function ReLU6(x) = min(max(0, x), 6) . Mirrors torch.nn.ReLU6; the kernel is the ATen hardtanh(0, 6) port. add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # RReLU URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.RReLU.html RReLU class tensorplay.nn.modules.activation. RReLU ( lower : float = 0.125 , upper : float = 0.3333333333333333 , inplace : bool = False ) [source] Applies the randomized leaky rectified linear unit function, element-wise. Method described in the paper: Empirical Evaluation of Rectified Activations in Convolutional Network . Parameters : lower – lower bound of the uniform distribution. Default: \(\frac{1}{8}\) upper – upper bound of the uniform distribution. Default: \(\frac{1}{3}\) inplace – can optionally do the operation in-place. Default: False add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # SELU URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.SELU.html SELU class tensorplay.nn.modules.activation. SELU ( inplace : bool = False ) [source] Applies selu with ATen’s fixed lambda/alpha constants. add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Sigmoid URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.Sigmoid.html Sigmoid class tensorplay.nn.modules.activation. Sigmoid ( * args , ** kwargs ) [source] Applies the Sigmoid function element-wise. \[\text{Sigmoid}(x) = \sigma(x) = \frac{1}{1 + \exp(-x)}\] Shape: Input: \((*)\) , where \(*\) means any number of dimensions. Output: \((*)\) , same shape as the input. Examples: >>> m = nn . Sigmoid () >>> input = tensorplay . randn ( 2 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # SiLU URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.SiLU.html SiLU class tensorplay.nn.modules.activation. SiLU ( inplace : bool = False ) [source] Applies the Sigmoid Linear Unit (SiLU) function, element-wise. The SiLU function is also known as the swish function. \[\text{silu}(x) = x * \sigma(x), \text{where } \sigma(x) \text{ is the logistic sigmoid.}\] Note See Gaussian Error Linear Units (GELUs) where the SiLU (Sigmoid Linear Unit) was originally coined, and see Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning and Swish: a Self-Gated Activation Function where the SiLU was experimented with later. Shape: Input: \((*)\) , where \(*\) means any number of dimensions. Output: \((*)\) , same shape as the input. Examples: >>> m = nn . SiLU () >>> input = tensorplay . randn ( 2 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str [source] Return the extra representation of the module. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Softmax URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.Softmax.html Softmax class tensorplay.nn.modules.activation. Softmax ( dim = None ) [source] Softmax over dim , mirroring torch.nn.Softmax. add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Softmin URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.Softmin.html Softmin class tensorplay.nn.modules.activation. Softmin ( dim : int | None = None ) [source] Applies the Softmin function to an n-dimensional input Tensor. Rescales them so that the elements of the n-dimensional output Tensor lie in the range [0, 1] and sum to 1. Softmin is defined as: \[\text{Softmin}(x_{i}) = \frac{\exp(-x_i)}{\sum_j \exp(-x_j)}\] Parameters : dim ( int ) – A dimension along which Softmin will be computed (so every slice along dim will sum to 1). add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Softplus URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.Softplus.html Softplus class tensorplay.nn.modules.activation. Softplus ( beta : float = 1.0 , threshold : float = 20.0 ) [source] Applies softplus with linearization above threshold * beta . add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Softshrink URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.Softshrink.html Softshrink class tensorplay.nn.modules.activation. Softshrink ( lambd : float = 0.5 ) [source] Applies the soft shrinkage function element-wise. \[\begin{split}\text{SoftShrinkage}(x) = \begin{cases} x - \lambda & \text{ if } x > \lambda \\ x + \lambda & \text{ if } x < -\lambda \\ 0 & \text{ otherwise } \end{cases}\end{split}\] Parameters : lambd – the \(\lambda\) (must be no less than zero) value for the Softshrink formulation. Default: 0.5 add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Softsign URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.Softsign.html Softsign class tensorplay.nn.modules.activation. Softsign ( * args , ** kwargs ) [source] Applies the element-wise function: \[\text{SoftSign}(x) = \frac{x}{ 1 + |x|}\] Shape: Input: \((*)\) , where \(*\) means any number of dimensions. Output: \((*)\) , same shape as the input. add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Tanh URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.Tanh.html Tanh class tensorplay.nn.modules.activation. Tanh ( * args , ** kwargs ) [source] Applies the Hyperbolic Tangent (Tanh) function element-wise. Tanh is defined as: \[\text{Tanh}(x) = \tanh(x) = \frac{\exp(x) - \exp(-x)} {\exp(x) + \exp(-x)}\] Shape: Input: \((*)\) , where \(*\) means any number of dimensions. Output: \((*)\) , same shape as the input. Examples: >>> m = nn . Tanh () >>> input = tensorplay . randn ( 2 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Tanhshrink URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.Tanhshrink.html Tanhshrink class tensorplay.nn.modules.activation. Tanhshrink ( * args , ** kwargs ) [source] Applies element-wise, \(\text{Tanhshrink}(x) = x - \text{Tanh}(x)\) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Threshold URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.activation.Threshold.html Threshold class tensorplay.nn.modules.activation. Threshold ( threshold : float , value : float , inplace : bool = False ) [source] Thresholds each element of the input Tensor. Threshold is defined as: \[\begin{split}y = \begin{cases} x, &\text{ if } x > \text{threshold} \\ \text{value}, &\text{ otherwise } \end{cases}\end{split}\] Parameters : threshold – The value to threshold at value – The value to replace with inplace – can optionally do the operation in-place. Default: False Shape: Input: \((*)\) , where \(*\) means any number of dimensions. Output: \((*)\) , same shape as the input. Examples: >>> m = tensorplay . nn . Threshold ( 0 , 0.5 ) >>> input = tensorplay . arange ( - 3 , 3 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # AdaptiveLogSoftmaxWithLoss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.adaptive.AdaptiveLogSoftmaxWithLoss.html AdaptiveLogSoftmaxWithLoss class tensorplay.nn.modules.adaptive. AdaptiveLogSoftmaxWithLoss ( in_features : int , n_classes : int , cutoffs : Sequence [ int ] , div_value : float = 4.0 , head_bias : bool = False , device = None , dtype = None ) [source] Efficient softmax approximation. As described in Efficient softmax approximation for GPUs by Edouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, and Hervé Jégou . Adaptive softmax is an approximate strategy for training models with large output spaces. It is most effective when the label distribution is highly imbalanced, for example in natural language modelling, where the word frequency distribution approximately follows the Zipf’s law . Adaptive softmax partitions the labels into several clusters, according to their frequency. These clusters may contain different number of targets each. Additionally, clusters containing less frequent labels assign lower dimensional embeddings to those labels, which speeds up the computation. For each minibatch, only clusters for which at least one target is present are evaluated. The idea is that the clusters which are accessed frequently (like the first one, containing most frequent labels), should also be cheap to compute – that is, contain a small number of assigned labels. We highly recommend taking a look at the original paper for more details. cutoffs should be an ordered Sequence of integers sorted in the increasing order. It controls number of clusters and the partitioning of targets into clusters. For example setting cutoffs = [10, 100, 1000] means that first 10 targets will be assigned to the ‘head’ of the adaptive softmax, targets 11, 12, …, 100 will be assigned to the first cluster, and targets 101, 102, …, 1000 will be assigned to the second cluster, while targets 1001, 1002, …, n_classes - 1 will be assigned to the last, third cluster. div_value is used to compute the size of each additional cluster, which is given as \(\left\lfloor\frac{\texttt{in\_features}}{\texttt{div\_value}^{idx}}\right\rfloor\) , where \(idx\) is the cluster index (with clusters for less frequent words having larger indices, and indices starting from \(1\) ). head_bias if set to True, adds a bias term to the ‘head’ of the adaptive softmax. See paper for details. Set to False in the official implementation. Warning Labels passed as inputs to this module should be sorted according to their frequency. This means that the most frequent label should be represented by the index 0 , and the least frequent label should be represented by the index n_classes - 1 . Note This module returns a NamedTuple with output and loss fields. See further documentation for details. Note To compute log-probabilities for all classes, the log_prob method can be used. Parameters : in_features ( int ) – Number of features in the input tensor n_classes ( int ) – Number of classes in the dataset cutoffs ( Sequence ) – Cutoffs used to assign targets to their buckets div_value ( float , optional ) – value used as an exponent to compute sizes of the clusters. Default: 4.0 head_bias ( bool , optional ) – If True , adds a bias term to the ‘head’ of the adaptive softmax. Default: False Returns : output is a Tensor of size N containing computed target log probabilities for each example loss is a Scalar representing the computed negative log likelihood loss Return type : NamedTuple with output and loss fields Shape: input: \((N, \texttt{in\_features})\) or \((\texttt{in\_features})\) target: \((N)\) or \(()\) where each value satisfies \(0 <= \texttt{target[i]} <= \texttt{n\_classes}\) output1: \((N)\) or \(()\) output2: Scalar add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input_ : TensorBase , target_ : TensorBase ) → _ASMoutput [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . log_prob ( input : TensorBase ) → TensorBase [source] Compute log probabilities for all \(\texttt{n\_classes}\) . Parameters : input ( Tensor ) – a minibatch of examples Returns : log-probabilities for each class \(c\) in range \(0 <= c <= \texttt{n\_classes}\) , where \(\texttt{n\_classes}\) is a parameter passed to AdaptiveLogSoftmaxWithLoss constructor. Shape: Input: \((N, \texttt{in\_features})\) Output: \((N, \texttt{n\_classes})\) modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) predict ( input : TensorBase ) → TensorBase [source] Return the class with the highest probability for each example in the input minibatch. This is equivalent to self.log_prob(input).argmax(dim=1) , but is more efficient in some cases. Parameters : input ( Tensor ) – a minibatch of examples Returns : a class with the highest probability for each example Return type : output (Tensor) Shape: Input: \((N, \texttt{in\_features})\) Output: \((N)\) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module reset_parameters ( ) → None [source] Resets parameters based on their initialization used in __init__ . set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # BatchNorm1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.batchnorm.BatchNorm1d.html BatchNorm1d class tensorplay.nn.modules.batchnorm. BatchNorm1d ( num_features : int , eps : float = 1e-05 , momentum : float | None = 0.1 , affine : bool = True , track_running_stats : bool = True , device = None , dtype = None ) [source] Applies Batch Normalization over a 2D or 3D input. Method described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift . \[y = \frac{x - \mathrm{E}[x]}{\sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta\] The mean and standard-deviation are calculated per-dimension over the mini-batches and \(\gamma\) and \(\beta\) are learnable parameter vectors of size C (where C is the number of features or channels of the input). By default, the elements of \(\gamma\) are set to 1 and the elements of \(\beta\) are set to 0. At train time in the forward pass, the variance is calculated via the biased estimator, equivalent to tensorplay.var(input, unbiased=False) . However, the value stored in the moving average of the variance is calculated via the unbiased estimator, equivalent to tensorplay.var(input, unbiased=True) . Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default momentum of 0.1. If track_running_stats is set to False , this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well. Note This momentum argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is \(\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t\) , where \(\hat{x}\) is the estimated statistic and \(x_t\) is the new observed value. Because the Batch Normalization is done over the C dimension, computing statistics on (N, L) slices, it’s common terminology to call this Temporal Batch Normalization. Parameters : num_features – number of features or channels \(C\) of the input eps – a value added to the denominator for numerical stability. Default: 1e-5 momentum – the value used for the running_mean and running_var computation. Can be set to None for cumulative moving average (i.e. simple average). Default: 0.1 affine – a boolean value that when set to True , this module has learnable affine parameters. Default: True track_running_stats – a boolean value that when set to True , this module tracks the running mean and variance, and when set to False , this module does not track such statistics, and initializes statistics buffers running_mean and running_var as None . When these buffers are None , this module always uses batch statistics. in both training and eval modes. Default: True Shape: Input: \((N, C)\) or \((N, C, L)\) , where \(N\) is the batch size, \(C\) is the number of features or channels, and \(L\) is the sequence length Output: \((N, C)\) or \((N, C, L)\) (same shape as input) Examples: >>> # With Learnable Parameters >>> m = nn . BatchNorm1d ( 100 ) >>> # Without Learnable Parameters >>> m = nn . BatchNorm1d ( 100 , affine = False ) >>> input = tensorplay . randn ( 20 , 100 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # BatchNorm3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.batchnorm.BatchNorm3d.html BatchNorm3d class tensorplay.nn.modules.batchnorm. BatchNorm3d ( num_features : int , eps : float = 1e-05 , momentum : float | None = 0.1 , affine : bool = True , track_running_stats : bool = True , device = None , dtype = None ) [source] Applies Batch Normalization over a 5D input. 5D is a mini-batch of 3D inputs with additional channel dimension as described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift . \[y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta\] The mean and standard-deviation are calculated per-dimension over the mini-batches and \(\gamma\) and \(\beta\) are learnable parameter vectors of size C (where C is the input size). By default, the elements of \(\gamma\) are set to 1 and the elements of \(\beta\) are set to 0. At train time in the forward pass, the standard-deviation is calculated via the biased estimator, equivalent to tensorplay.var(input, unbiased=False) . However, the value stored in the moving average of the standard-deviation is calculated via the unbiased estimator, equivalent to tensorplay.var(input, unbiased=True) . Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default momentum of 0.1. If track_running_stats is set to False , this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well. Note This momentum argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is \(\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t\) , where \(\hat{x}\) is the estimated statistic and \(x_t\) is the new observed value. Because the Batch Normalization is done over the C dimension, computing statistics on (N, D, H, W) slices, it’s common terminology to call this Volumetric Batch Normalization or Spatio-temporal Batch Normalization. Parameters : num_features – \(C\) from an expected input of size \((N, C, D, H, W)\) eps – a value added to the denominator for numerical stability. Default: 1e-5 momentum – the value used for the running_mean and running_var computation. Can be set to None for cumulative moving average (i.e. simple average). Default: 0.1 affine – a boolean value that when set to True , this module has learnable affine parameters. Default: True track_running_stats – a boolean value that when set to True , this module tracks the running mean and variance, and when set to False , this module does not track such statistics, and initializes statistics buffers running_mean and running_var as None . When these buffers are None , this module always uses batch statistics. in both training and eval modes. Default: True Shape: Input: \((N, C, D, H, W)\) Output: \((N, C, D, H, W)\) (same shape as input) Examples: >>> # With Learnable Parameters >>> m = nn . BatchNorm3d ( 100 ) >>> # Without Learnable Parameters >>> m = nn . BatchNorm3d ( 100 , affine = False ) >>> input = tensorplay . randn ( 20 , 100 , 35 , 45 , 10 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # LazyBatchNorm1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.batchnorm.LazyBatchNorm1d.html LazyBatchNorm1d class tensorplay.nn.modules.batchnorm. LazyBatchNorm1d ( eps = 1e-05 , momentum = 0.1 , affine = True , track_running_stats = True , device = None , dtype = None ) [source] A tensorplay.nn.BatchNorm1d module with lazy initialization. Lazy initialization based on the num_features argument of the BatchNorm1d that is inferred from the input.size(1) . The attributes that will be lazily initialized are weight , bias , running_mean and running_var . Check the tensorplay.nn.modules.lazy.LazyModuleMixin for further documentation on lazy modules and their limitations. Parameters : eps – a value added to the denominator for numerical stability. Default: 1e-5 momentum – the value used for the running_mean and running_var computation. Can be set to None for cumulative moving average (i.e. simple average). Default: 0.1 affine – a boolean value that when set to True , this module has learnable affine parameters. Default: True track_running_stats – a boolean value that when set to True , this module tracks the running mean and variance, and when set to False , this module does not track such statistics, and initializes statistics buffers running_mean and running_var as None . When these buffers are None , this module always uses batch statistics. in both training and eval modes. Default: True add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module cls_to_become alias of BatchNorm1d compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module has_uninitialized_params ( ) Check if a module has parameters that are not initialized. load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # LazyBatchNorm2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.batchnorm.LazyBatchNorm2d.html LazyBatchNorm2d class tensorplay.nn.modules.batchnorm. LazyBatchNorm2d ( eps = 1e-05 , momentum = 0.1 , affine = True , track_running_stats = True , device = None , dtype = None ) [source] A tensorplay.nn.BatchNorm2d module with lazy initialization. Lazy initialization is done for the num_features argument of the BatchNorm2d that is inferred from the input.size(1) . The attributes that will be lazily initialized are weight , bias , running_mean and running_var . Check the tensorplay.nn.modules.lazy.LazyModuleMixin for further documentation on lazy modules and their limitations. Parameters : eps – a value added to the denominator for numerical stability. Default: 1e-5 momentum – the value used for the running_mean and running_var computation. Can be set to None for cumulative moving average (i.e. simple average). Default: 0.1 affine – a boolean value that when set to True , this module has learnable affine parameters. Default: True track_running_stats – a boolean value that when set to True , this module tracks the running mean and variance, and when set to False , this module does not track such statistics, and initializes statistics buffers running_mean and running_var as None . When these buffers are None , this module always uses batch statistics. in both training and eval modes. Default: True add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module cls_to_become alias of BatchNorm2d compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module has_uninitialized_params ( ) Check if a module has parameters that are not initialized. load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # LazyBatchNorm3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.batchnorm.LazyBatchNorm3d.html LazyBatchNorm3d class tensorplay.nn.modules.batchnorm. LazyBatchNorm3d ( eps = 1e-05 , momentum = 0.1 , affine = True , track_running_stats = True , device = None , dtype = None ) [source] A tensorplay.nn.BatchNorm3d module with lazy initialization. Lazy initialization is done for the num_features argument of the BatchNorm3d that is inferred from the input.size(1) . The attributes that will be lazily initialized are weight , bias , running_mean and running_var . Check the tensorplay.nn.modules.lazy.LazyModuleMixin for further documentation on lazy modules and their limitations. Parameters : eps – a value added to the denominator for numerical stability. Default: 1e-5 momentum – the value used for the running_mean and running_var computation. Can be set to None for cumulative moving average (i.e. simple average). Default: 0.1 affine – a boolean value that when set to True , this module has learnable affine parameters. Default: True track_running_stats – a boolean value that when set to True , this module tracks the running mean and variance, and when set to False , this module does not track such statistics, and initializes statistics buffers running_mean and running_var as None . When these buffers are None , this module always uses batch statistics. in both training and eval modes. Default: True add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module cls_to_become alias of BatchNorm3d compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module has_uninitialized_params ( ) Check if a module has parameters that are not initialized. load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # SyncBatchNorm URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.batchnorm.SyncBatchNorm.html SyncBatchNorm class tensorplay.nn.modules.batchnorm. SyncBatchNorm ( num_features : int , eps : float = 1e-05 , momentum : float | None = 0.1 , affine : bool = True , track_running_stats : bool = True , process_group = None , device = None , dtype = None ) [source] Applies Batch Normalization over a N-Dimensional input with synchronized batch statistics across all processes in the group. See BatchNorm2d (and torch’s SyncBatchNorm ) for the semantics; during training the per-rank mean/invstd are gathered and combined with a count-weighted parallel-variance formula, so every rank normalizes with identical global statistics. Eval mode and single-process runs fall back to plain BatchNorm behavior. Currently only DistributedDataParallel usage is supported. Use convert_sync_batchnorm() to convert BatchNorm*d layers before wrapping the module in DDP. Parameters : num_features – \(C\) from an expected input of size \((N, C, +)\) eps – a value added to the denominator for numerical stability. Default: 1e-5 momentum – the value used for the running_mean and running_var computation. Can be None for cumulative moving average. Default: 0.1 affine – a boolean value that when set to True , this module has learnable affine parameters. Default: True track_running_stats – whether to track running statistics. Default: True process_group – process group over which statistics are synchronized. Default: the whole world. add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. classmethod convert_sync_batchnorm ( module , process_group = None ) [source] Converts all BatchNorm*d layers in the model to SyncBatchNorm layers. Parameters : module ( nn.Module ) – module containing one or more BatchNorm*d layers process_group ( optional ) – process group to scope synchronization, default is the whole world Returns : The original module with converted SyncBatchNorm layers. If the original module is a BatchNorm*d layer, a new SyncBatchNorm layer object will be returned instead. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # ChannelShuffle URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.channelshuffle.ChannelShuffle.html ChannelShuffle class tensorplay.nn.modules.channelshuffle. ChannelShuffle ( groups : int ) [source] Divides and rearranges the channels in a tensor. This operation divides the channels in a tensor of shape \((N, C, *)\) into g groups as \((N, \frac{C}{g}, g, *)\) and shuffles them, while retaining the original tensor shape in the final output. Parameters : groups ( int ) – number of groups to divide channels in. Examples: >>> channel_shuffle = nn . ChannelShuffle ( 2 ) >>> input = torch . arange ( 1 , 17 , dtype = torch . float32 ) . view ( 1 , 4 , 2 , 2 ) >>> input tensor([[[[ 1., 2.], [ 3., 4.]], [[ 5., 6.], [ 7., 8.]], [[ 9., 10.], [11., 12.]], [[13., 14.], [15., 16.]]]]) >>> output = channel_shuffle ( input ) >>> output tensor([[[[ 1., 2.], [ 3., 4.]], [[ 9., 10.], [11., 12.]], [[ 5., 6.], [ 7., 8.]], [[13., 14.], [15., 16.]]]]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str [source] Return the extra representation of the module. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # ModuleDict URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.container.ModuleDict.html ModuleDict class tensorplay.nn.modules.container. ModuleDict ( modules : Mapping [ str , Module ] | None = None ) [source] Holds submodules in a dictionary. ModuleDict can be indexed like a regular Python dictionary, but modules it contains are properly registered, and will be visible by all Module methods. ModuleDict is an ordered dictionary that respects the order of insertion, and in update() , the order of the merged OrderedDict , dict (started from Python 3.6) or another ModuleDict (the argument to update() ). Note that update() with other unordered mapping types (e.g., Python’s plain dict before Python version 3.6) does not preserve the order of the merged mapping. Parameters : modules ( iterable , optional ) – a mapping (dictionary) of (string: module) or an iterable of key-value pairs of type (string, module) Example: class MyModule ( nn . Module ): def __init__ ( self ) -> None : super () . __init__ () self . choices = nn . ModuleDict ( { "conv" : nn . Conv2d ( 10 , 10 , 3 ), "pool" : nn . MaxPool2d ( 3 )} ) self . activations = nn . ModuleDict ( [[ "lrelu" , nn . LeakyReLU ()], [ "prelu" , nn . PReLU ()]] ) def forward ( self , x , choice , act ): x = self . choices [ choice ]( x ) x = self . activations [ act ]( x ) return x add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module clear ( ) → None [source] Remove all items from the ModuleDict. compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( * input : Any ) → None Define the computation performed at every call. Should be overridden by all subclasses. Note Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module items ( ) → ItemsView [ str , Module ] [source] Return an iterable of the ModuleDict key/value pairs. keys ( ) → KeysView [ str ] [source] Return an iterable of the ModuleDict keys. load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) pop ( key : str ) → Module [source] Remove key from the ModuleDict and return its module. Parameters : key ( str ) – key to pop from the ModuleDict register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module update ( modules : Mapping [ str , Module ] ) → None [source] Update the ModuleDict with key-value pairs from a mapping, overwriting existing keys. Note If modules is an OrderedDict , a ModuleDict , or an iterable of key-value pairs, the order of new elements in it is preserved. Parameters : modules ( iterable ) – a mapping (dictionary) from string to Module , or an iterable of key-value pairs of type (string, Module ) values ( ) → ValuesView [ Module ] [source] Return an iterable of the ModuleDict values. zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # ModuleList URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.container.ModuleList.html ModuleList class tensorplay.nn.modules.container. ModuleList ( modules : Iterable [ Module ] | None = None ) [source] Holds submodules in a list. ModuleList can be indexed like a regular Python list, but modules it contains are properly registered, and will be visible by all Module methods. Parameters : modules ( iterable , optional ) – an iterable of modules to add Example: class MyModule ( nn . Module ): def __init__ ( self ) -> None : super () . __init__ () self . linear = nn . ModuleList ([ nn . Linear ( 10 , 10 ) for i in range ( 10 )]) def forward ( self , x ): # ModuleList can act as an iterable, or be indexed using ints for i , l in enumerate ( self . linear ): x = self . linear [ i // 2 ]( x ) + l ( x ) return x add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. append ( module : Module ) → Self [source] Append a given module to the end of the list. Parameters : module ( nn.Module ) – module to append apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extend ( modules : Iterable [ Module ] ) → Self [source] Append modules from a Python iterable to the end of the list. Parameters : modules ( iterable ) – iterable of modules to append extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( * input : Any ) → None Define the computation performed at every call. Should be overridden by all subclasses. Note Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module insert ( index : int , module : Module ) → None [source] Insert a given module before a given index in the list. Parameters : index ( int ) – index to insert. module ( nn.Module ) – module to insert load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # ParameterDict URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.container.ParameterDict.html ParameterDict class tensorplay.nn.modules.container. ParameterDict ( parameters : Any | None = None ) [source] Holds parameters in a dictionary. ParameterDict can be indexed like a regular Python dictionary, but Parameters it contains are properly registered, and will be visible by all Module methods. Other objects are treated as would be done by a regular Python dictionary ParameterDict is an ordered dictionary. update() with other unordered mapping types (e.g., Python’s plain dict ) does not preserve the order of the merged mapping. On the other hand, OrderedDict or another ParameterDict will preserve their ordering. Note that the constructor, assigning an element of the dictionary and the update() method will convert any Tensor into Parameter . Parameters : values ( iterable , optional ) – a mapping (dictionary) of (string : Any) or an iterable of key-value pairs of type (string, Any) Example: class MyModule ( nn . Module ): def __init__ ( self ) -> None : super () . __init__ () self . params = nn . ParameterDict ( { "left" : nn . Parameter ( tensorplay . randn ( 5 , 10 )), "right" : nn . Parameter ( tensorplay . randn ( 5 , 10 )), } ) def forward ( self , x , choice ): x = self . params [ choice ] . mm ( x ) return x add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module clear ( ) → None [source] Remove all items from the ParameterDict. compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. copy ( ) → ParameterDict [source] Return a copy of this ParameterDict instance. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( * input : Any ) → None Define the computation performed at every call. Should be overridden by all subclasses. Note Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them. static fromkeys ( keys : Iterable [ str ] , default : Any | None = None ) → ParameterDict [source] Return a new ParameterDict with the keys provided. Parameters : keys ( iterable , string ) – keys to make the new ParameterDict from default ( Parameter , optional ) – value to set for all keys get ( key : str , default : Any | None = None ) → Any [source] Return the parameter associated with key if present. Otherwise return default if provided, None if not. Parameters : key ( str ) – key to get from the ParameterDict default ( Parameter , optional ) – value to return if key not present get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module items ( ) → Iterable [ tuple [ str , Any ] ] [source] Return an iterable of the ParameterDict key/value pairs. keys ( ) → KeysView [ str ] [source] Return an iterable of the ParameterDict keys. load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) pop ( key : str ) → Any [source] Remove key from the ParameterDict and return its parameter. Parameters : key ( str ) – key to pop from the ParameterDict popitem ( ) → tuple [ str , Any ] [source] Remove and return the last inserted (key, parameter) pair from the ParameterDict. register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . setdefault ( key : str , default : Any | None = None ) → Any [source] Set the default for a key in the ParameterDict. If key is in the ParameterDict, return its value. If not, insert key with a parameter default and return default . default defaults to None . Parameters : key ( str ) – key to set default for default ( Any ) – the parameter set to the key share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module update ( parameters : Mapping [ str , Any ] | ParameterDict ) → None [source] Update the ParameterDict with key-value pairs from parameters , overwriting existing keys. Note If parameters is an OrderedDict , a ParameterDict , or an iterable of key-value pairs, the order of new elements in it is preserved. Parameters : parameters ( iterable ) – a mapping (dictionary) from string to Parameter , or an iterable of key-value pairs of type (string, Parameter ) values ( ) → Iterable [ Any ] [source] Return an iterable of the ParameterDict values. zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # ParameterList URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.container.ParameterList.html ParameterList class tensorplay.nn.modules.container. ParameterList ( values : Iterable [ Any ] | None = None ) [source] Holds parameters in a list. ParameterList can be used like a regular Python list, but Tensors that are Parameter are properly registered, and will be visible by all Module methods. Note that the constructor, assigning an element of the list, the append() method and the extend() method will convert any Tensor into Parameter . Parameters : values ( iterable , optional ) – an iterable of elements to add to the list. Example: class MyModule ( nn . Module ): def __init__ ( self ) -> None : super () . __init__ () self . params = nn . ParameterList ( [ nn . Parameter ( tensorplay . randn ( 10 , 10 )) for i in range ( 10 )] ) def forward ( self , x ): # ParameterList can act as an iterable, or be indexed using ints for i , p in enumerate ( self . params ): x = self . params [ i // 2 ] . mm ( x ) + p . mm ( x ) return x add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. append ( value : Any ) → Self [source] Append a given value at the end of the list. Parameters : value ( Any ) – value to append apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extend ( values : Iterable [ Any ] ) → Self [source] Append values from a Python iterable to the end of the list. Parameters : values ( iterable ) – iterable of values to append extra_repr ( ) → str [source] Return the extra representation of the module. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( * input : Any ) → None Define the computation performed at every call. Should be overridden by all subclasses. Note Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Sequential URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.container.Sequential.html Sequential class tensorplay.nn.modules.container. Sequential ( * args : Module ) [source] class tensorplay.nn.modules.container. Sequential ( arg : OrderedDict [ str , Module ] ) A sequential container. Modules will be added to it in the order they are passed in the constructor. Alternatively, an OrderedDict of modules can be passed in. The forward() method of Sequential accepts any input and forwards it to the first module it contains. It then “chains” outputs to inputs sequentially for each subsequent module, finally returning the output of the last module. The value a Sequential provides over manually calling a sequence of modules is that it allows treating the whole container as a single module, such that performing a transformation on the Sequential applies to each of the modules it stores (which are each a registered submodule of the Sequential ). What’s the difference between a Sequential and a tensorplay.nn.ModuleList ? A ModuleList is exactly what it sounds like–a list for storing Module s! On the other hand, the layers in a Sequential are connected in a cascading way. Example: # Using Sequential to create a small model. When `model` is run, # input will first be passed to `Conv2d(1,20,5)`. The output of # `Conv2d(1,20,5)` will be used as the input to the first # `ReLU`; the output of the first `ReLU` will become the input # for `Conv2d(20,64,5)`. Finally, the output of # `Conv2d(20,64,5)` will be used as input to the second `ReLU` model = nn . Sequential ( nn . Conv2d ( 1 , 20 , 5 ), nn . ReLU (), nn . Conv2d ( 20 , 64 , 5 ), nn . ReLU () ) # Using Sequential with OrderedDict. This is functionally the # same as the above code model = nn . Sequential ( OrderedDict ( [ ( "conv1" , nn . Conv2d ( 1 , 20 , 5 )), ( "relu1" , nn . ReLU ()), ( "conv2" , nn . Conv2d ( 20 , 64 , 5 )), ( "relu2" , nn . ReLU ()), ] ) ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. append ( module : Module ) → Self [source] Append a given module to the end. Parameters : module ( nn.Module ) – module to append Example: >>> import tensorplay.nn as nn >>> n = nn . Sequential ( nn . Linear ( 1 , 2 ), nn . Linear ( 2 , 3 )) >>> n . append ( nn . Linear ( 3 , 4 )) Sequential( (0): Linear(in_features=1, out_features=2, bias=True) (1): Linear(in_features=2, out_features=3, bias=True) (2): Linear(in_features=3, out_features=4, bias=True) ) apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extend ( sequential : Iterable [ Module ] ) → Self [source] Extends the current Sequential container with layers from another Sequential container. Parameters : sequential ( Sequential ) – A Sequential container whose layers will be added to the current container. Example: >>> import tensorplay.nn as nn >>> n = nn . Sequential ( nn . Linear ( 1 , 2 ), nn . Linear ( 2 , 3 )) >>> other = nn . Sequential ( nn . Linear ( 3 , 4 ), nn . Linear ( 4 , 5 )) >>> n . extend ( other ) # or `n + other` Sequential( (0): Linear(in_features=1, out_features=2, bias=True) (1): Linear(in_features=2, out_features=3, bias=True) (2): Linear(in_features=3, out_features=4, bias=True) (3): Linear(in_features=4, out_features=5, bias=True) ) extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input ) [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module insert ( index : int , module : Module ) → Self [source] Inserts a module into the Sequential container at the specified index. Parameters : index ( int ) – The index to insert the module. module ( Module ) – The module to be inserted. Example: >>> import tensorplay.nn as nn >>> n = nn . Sequential ( nn . Linear ( 1 , 2 ), nn . Linear ( 2 , 3 )) >>> n . insert ( 0 , nn . Linear ( 3 , 4 )) Sequential( (0): Linear(in_features=3, out_features=4, bias=True) (1): Linear(in_features=1, out_features=2, bias=True) (2): Linear(in_features=2, out_features=3, bias=True) ) load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) pop ( key : int | slice ) → Module [source] Pop key from self. register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Conv1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.conv.Conv1d.html Conv1d class tensorplay.nn.modules.conv. Conv1d ( in_channels , out_channels , kernel_size , stride = 1 , padding = 0 , dilation = 1 , groups = 1 , bias = True , padding_mode = 'zeros' , device = None , dtype = None ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Conv2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.conv.Conv2d.html Conv2d class tensorplay.nn.modules.conv. Conv2d ( in_channels , out_channels , kernel_size , stride = 1 , padding = 0 , dilation = 1 , groups = 1 , bias = True , padding_mode = 'zeros' , device = None , dtype = None ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Conv3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.conv.Conv3d.html Conv3d class tensorplay.nn.modules.conv. Conv3d ( in_channels , out_channels , kernel_size , stride = 1 , padding = 0 , dilation = 1 , groups = 1 , bias = True , padding_mode = 'zeros' , device = None , dtype = None ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # ConvTranspose1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.conv.ConvTranspose1d.html ConvTranspose1d class tensorplay.nn.modules.conv. ConvTranspose1d ( in_channels , out_channels , kernel_size , stride = 1 , padding = 0 , output_padding = 0 , groups = 1 , bias = True , dilation = 1 , padding_mode = 'zeros' , device = None , dtype = None ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # ConvTranspose2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.conv.ConvTranspose2d.html ConvTranspose2d class tensorplay.nn.modules.conv. ConvTranspose2d ( in_channels , out_channels , kernel_size , stride = 1 , padding = 0 , output_padding = 0 , groups = 1 , bias = True , dilation = 1 , padding_mode = 'zeros' , device = None , dtype = None ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # ConvTranspose3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.conv.ConvTranspose3d.html ConvTranspose3d class tensorplay.nn.modules.conv. ConvTranspose3d ( in_channels , out_channels , kernel_size , stride = 1 , padding = 0 , output_padding = 0 , groups = 1 , bias = True , dilation = 1 , padding_mode = 'zeros' , device = None , dtype = None ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # LazyConv1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.conv.LazyConv1d.html LazyConv1d class tensorplay.nn.modules.conv. LazyConv1d ( out_channels , kernel_size , stride = 1 , padding = 0 , dilation = 1 , groups = 1 , bias = True , padding_mode = 'zeros' , device = None , dtype = None ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module has_uninitialized_params ( ) Check if a module has parameters that are not initialized. load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # LazyConv2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.conv.LazyConv2d.html LazyConv2d class tensorplay.nn.modules.conv. LazyConv2d ( out_channels , kernel_size , stride = 1 , padding = 0 , dilation = 1 , groups = 1 , bias = True , padding_mode = 'zeros' , device = None , dtype = None ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module has_uninitialized_params ( ) Check if a module has parameters that are not initialized. load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # LazyConv3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.conv.LazyConv3d.html LazyConv3d class tensorplay.nn.modules.conv. LazyConv3d ( out_channels , kernel_size , stride = 1 , padding = 0 , dilation = 1 , groups = 1 , bias = True , padding_mode = 'zeros' , device = None , dtype = None ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module has_uninitialized_params ( ) Check if a module has parameters that are not initialized. load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # LazyConvTranspose1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.conv.LazyConvTranspose1d.html LazyConvTranspose1d class tensorplay.nn.modules.conv. LazyConvTranspose1d ( out_channels , kernel_size , stride = 1 , padding = 0 , output_padding = 0 , groups = 1 , bias = True , dilation = 1 , padding_mode = 'zeros' , device = None , dtype = None ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module has_uninitialized_params ( ) Check if a module has parameters that are not initialized. load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # LazyConvTranspose2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.conv.LazyConvTranspose2d.html LazyConvTranspose2d class tensorplay.nn.modules.conv. LazyConvTranspose2d ( out_channels , kernel_size , stride = 1 , padding = 0 , output_padding = 0 , groups = 1 , bias = True , dilation = 1 , padding_mode = 'zeros' , device = None , dtype = None ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module has_uninitialized_params ( ) Check if a module has parameters that are not initialized. load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # LazyConvTranspose3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.conv.LazyConvTranspose3d.html LazyConvTranspose3d class tensorplay.nn.modules.conv. LazyConvTranspose3d ( out_channels , kernel_size , stride = 1 , padding = 0 , output_padding = 0 , groups = 1 , bias = True , dilation = 1 , padding_mode = 'zeros' , device = None , dtype = None ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module has_uninitialized_params ( ) Check if a module has parameters that are not initialized. load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # CosineSimilarity URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.distance.CosineSimilarity.html CosineSimilarity class tensorplay.nn.modules.distance. CosineSimilarity ( dim : int = 1 , eps : float = 1e-08 ) [source] Returns cosine similarity between \(x_1\) and \(x_2\) , computed along dim . \[\text{similarity} = \dfrac{x_1 \cdot x_2}{\max(\Vert x_1 \Vert _2 \cdot \Vert x_2 \Vert _2, \epsilon)}.\] Parameters : dim ( int , optional ) – Dimension where cosine similarity is computed. Default: 1 eps ( float , optional ) – Small value to avoid division by zero. Default: 1e-8 Shape: Input1: \((\ast_1, D, \ast_2)\) where D is at position dim Input2: \((\ast_1, D, \ast_2)\) , same number of dimensions as x1, matching x1 size at dimension dim , and broadcastable with x1 at other dimensions. Output: \((\ast_1, \ast_2)\) Examples >>> input1 = torch . randn ( 100 , 128 ) >>> input2 = torch . randn ( 100 , 128 ) >>> cos = nn . CosineSimilarity ( dim = 1 , eps = 1e-6 ) >>> output = cos ( input1 , input2 ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( x1 : TensorBase , x2 : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # PairwiseDistance URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.distance.PairwiseDistance.html PairwiseDistance class tensorplay.nn.modules.distance. PairwiseDistance ( p : float = 2.0 , eps : float = 1e-06 , keepdim : bool = False ) [source] Computes the pairwise distance between input vectors, or between columns of input matrices. Distances are computed using p -norm, with constant eps added to avoid division by zero if p is negative, i.e.: \[\mathrm{dist}\left(x, y\right) = \left\Vert x-y + \epsilon e \right\Vert_p,\] where \(e\) is the vector of ones and the p -norm is given by. \[\Vert x \Vert _p = \left( \sum_{i=1}^n \vert x_i \vert ^ p \right) ^ {1/p}.\] Parameters : p ( real , optional ) – the norm degree. Can be negative. Default: 2 eps ( float , optional ) – Small value to avoid division by zero. Default: 1e-6 keepdim ( bool , optional ) – Determines whether or not to keep the vector dimension. Default: False Shape: Input1: \((N, D)\) or \((D)\) where N = batch dimension and D = vector dimension Input2: \((N, D)\) or \((D)\) , same shape as the Input1 Output: \((N)\) or \(()\) based on input dimension. If keepdim is True , then \((N, 1)\) or \((1)\) based on input dimension. Examples >>> pdist = nn . PairwiseDistance ( p = 2 ) >>> input1 = torch . randn ( 100 , 128 ) >>> input2 = torch . randn ( 100 , 128 ) >>> output = pdist ( input1 , input2 ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( x1 : TensorBase , x2 : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # AlphaDropout URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.dropout.AlphaDropout.html AlphaDropout class tensorplay.nn.modules.dropout. AlphaDropout ( p : float = 0.5 , inplace : bool = False ) [source] Applies Alpha Dropout over the input. Alpha Dropout is a type of Dropout that maintains the self-normalizing property. For an input with zero mean and unit standard deviation, the output of Alpha Dropout maintains the original mean and standard deviation of the input. Alpha Dropout goes hand-in-hand with SELU activation function, which ensures that the outputs have zero mean and unit standard deviation. During training, it randomly masks some of the elements of the input tensor with probability p using samples from a bernoulli distribution. The elements to masked are randomized on every forward call, and scaled and shifted to maintain zero mean and unit standard deviation. During evaluation the module simply computes an identity function. More details can be found in the paper Self-Normalizing Neural Networks . Parameters : p ( float ) – probability of an element to be dropped. Default: 0.5 inplace ( bool , optional ) – If set to True , will do this operation in-place Shape: Input: \((*)\) . Input can be of any shape Output: \((*)\) . Output is of the same shape as input Examples: >>> m = nn . AlphaDropout ( p = 0.2 ) >>> input = tensorplay . randn ( 20 , 16 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Dropout URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.dropout.Dropout.html Dropout class tensorplay.nn.modules.dropout. Dropout ( p : float = 0.5 , inplace : bool = False ) [source] During training, randomly zeroes some of the elements of the input tensor with probability p . The zeroed elements are chosen independently for each forward call and are sampled from a Bernoulli distribution. Each channel will be zeroed out independently on every forward call. This has proven to be an effective technique for regularization and preventing the co-adaptation of neurons as described in the paper Improving neural networks by preventing co-adaptation of feature detectors . Furthermore, the outputs are scaled by a factor of \(\frac{1}{1-p}\) during training. This means that during evaluation the module simply computes an identity function. Parameters : p – probability of an element to be zeroed. Default: 0.5 inplace – If set to True , will do this operation in-place. Default: False Shape: Input: \((*)\) . Input can be of any shape Output: \((*)\) . Output is of the same shape as input Examples: >>> m = nn . Dropout ( p = 0.2 ) >>> input = tensorplay . randn ( 20 , 16 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Dropout1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.dropout.Dropout1d.html Dropout1d class tensorplay.nn.modules.dropout. Dropout1d ( p : float = 0.5 , inplace : bool = False ) [source] Randomly zero out entire channels. A channel is a 1D feature map, e.g., the \(j\) -th channel of the \(i\) -th sample in the batched input is a 1D tensor \(\text{input}[i, j]\) . Each channel will be zeroed out independently on every forward call with probability p using samples from a Bernoulli distribution. Usually the input comes from nn.Conv1d modules. As described in the paper Efficient Object Localization Using Convolutional Networks , if adjacent pixels within feature maps are strongly correlated (as is normally the case in early convolution layers) then i.i.d. dropout will not regularize the activations and will otherwise just result in an effective learning rate decrease. In this case, nn.Dropout1d() will help promote independence between feature maps and should be used instead. Parameters : p ( float , optional ) – probability of an element to be zero-ed. inplace ( bool , optional ) – If set to True , will do this operation in-place Shape: Input: \((N, C, L)\) or \((C, L)\) . Output: \((N, C, L)\) or \((C, L)\) (same shape as input). Examples: >>> m = nn . Dropout1d ( p = 0.2 ) >>> input = tensorplay . randn ( 20 , 16 , 32 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Dropout2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.dropout.Dropout2d.html Dropout2d class tensorplay.nn.modules.dropout. Dropout2d ( p : float = 0.5 , inplace : bool = False ) [source] Randomly zero out entire channels. A channel is a 2D feature map, e.g., the \(j\) -th channel of the \(i\) -th sample in the batched input is a 2D tensor \(\text{input}[i, j]\) . Each channel will be zeroed out independently on every forward call with probability p using samples from a Bernoulli distribution. Usually the input comes from nn.Conv2d modules. As described in the paper Efficient Object Localization Using Convolutional Networks , if adjacent pixels within feature maps are strongly correlated (as is normally the case in early convolution layers) then i.i.d. dropout will not regularize the activations and will otherwise just result in an effective learning rate decrease. In this case, nn.Dropout2d() will help promote independence between feature maps and should be used instead. Parameters : p ( float , optional ) – probability of an element to be zero-ed. inplace ( bool , optional ) – If set to True , will do this operation in-place Warning Due to historical reasons, this class will perform 1D channel-wise dropout for 3D inputs (as done by nn.Dropout1d ). Thus, it currently does NOT support inputs without a batch dimension of shape \((C, H, W)\) . This behavior will change in a future release to interpret 3D inputs as no-batch-dim inputs. To maintain the old behavior, switch to nn.Dropout1d . Shape: Input: \((N, C, H, W)\) or \((N, C, L)\) . Output: \((N, C, H, W)\) or \((N, C, L)\) (same shape as input). Examples: >>> m = nn . Dropout2d ( p = 0.2 ) >>> input = tensorplay . randn ( 20 , 16 , 32 , 32 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Dropout3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.dropout.Dropout3d.html Dropout3d class tensorplay.nn.modules.dropout. Dropout3d ( p : float = 0.5 , inplace : bool = False ) [source] Randomly zero out entire channels. A channel is a 3D feature map, e.g., the \(j\) -th channel of the \(i\) -th sample in the batched input is a 3D tensor \(\text{input}[i, j]\) . Each channel will be zeroed out independently on every forward call with probability p using samples from a Bernoulli distribution. Usually the input comes from nn.Conv3d modules. As described in the paper Efficient Object Localization Using Convolutional Networks , if adjacent pixels within feature maps are strongly correlated (as is normally the case in early convolution layers) then i.i.d. dropout will not regularize the activations and will otherwise just result in an effective learning rate decrease. In this case, nn.Dropout3d() will help promote independence between feature maps and should be used instead. Parameters : p ( float , optional ) – probability of an element to be zeroed. inplace ( bool , optional ) – If set to True , will do this operation in-place Shape: Input: \((N, C, D, H, W)\) or \((C, D, H, W)\) . Output: \((N, C, D, H, W)\) or \((C, D, H, W)\) (same shape as input). Examples: >>> m = nn . Dropout3d ( p = 0.2 ) >>> input = tensorplay . randn ( 20 , 16 , 4 , 32 , 32 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # FeatureAlphaDropout URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.dropout.FeatureAlphaDropout.html FeatureAlphaDropout class tensorplay.nn.modules.dropout. FeatureAlphaDropout ( p : float = 0.5 , inplace : bool = False ) [source] Randomly masks out entire channels. A channel is a feature map, e.g. the \(j\) -th channel of the \(i\) -th sample in the batch input is a tensor \(\text{input}[i, j]\) of the input tensor). Instead of setting activations to zero, as in regular Dropout, the activations are set to the negative saturation value of the SELU activation function. More details can be found in the paper Self-Normalizing Neural Networks . Each element will be masked independently for each sample on every forward call with probability p using samples from a Bernoulli distribution. The elements to be masked are randomized on every forward call, and scaled and shifted to maintain zero mean and unit variance. Usually the input comes from nn.AlphaDropout modules. As described in the paper Efficient Object Localization Using Convolutional Networks , if adjacent pixels within feature maps are strongly correlated (as is normally the case in early convolution layers) then i.i.d. dropout will not regularize the activations and will otherwise just result in an effective learning rate decrease. In this case, nn.AlphaDropout() will help promote independence between feature maps and should be used instead. Parameters : p ( float , optional ) – probability of an element to be zeroed. Default: 0.5 inplace ( bool , optional ) – If set to True , will do this operation in-place Shape: Input: \((N, C, D, H, W)\) or \((C, D, H, W)\) . Output: \((N, C, D, H, W)\) or \((C, D, H, W)\) (same shape as input). Examples: >>> m = nn . FeatureAlphaDropout ( p = 0.2 ) >>> input = tensorplay . randn ( 20 , 16 , 4 , 32 , 32 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Flatten URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.flatten.Flatten.html Flatten class tensorplay.nn.modules.flatten. Flatten ( start_dim : int = 1 , end_dim : int = -1 ) [source] Flattens a contiguous range of dims into a tensor. For use with Sequential , see tensorplay.flatten() for details. Shape: Input: \((*, S_{\text{start}},..., S_{i}, ..., S_{\text{end}}, *)\) ,’ where \(S_{i}\) is the size at dimension \(i\) and \(*\) means any number of dimensions including none. Output: \((*, \prod_{i=\text{start}}^{\text{end}} S_{i}, *)\) . Parameters : start_dim – first dim to flatten (default = 1). end_dim – last dim to flatten (default = -1). Examples:: >>> input = tensorplay . randn ( 32 , 1 , 5 , 5 ) >>> # With default parameters >>> m = nn . Flatten () >>> output = m ( input ) >>> output . size () tensorplay.Size([32, 25]) >>> # With non-default parameters >>> m = nn . Flatten ( 0 , 2 ) >>> output = m ( input ) >>> output . size () tensorplay.Size([160, 5]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str [source] Returns the extra representation of the module. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Unflatten URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.flatten.Unflatten.html Unflatten class tensorplay.nn.modules.flatten. Unflatten ( dim : int | str , unflattened_size : Size | list [ int ] | tuple [ int , ... ] | tuple [ tuple [ str , int ] ] ) [source] Unflattens a tensor dim expanding it to a desired shape. For use with Sequential . dim specifies the dimension of the input tensor to be unflattened, and it can be either int or str when Tensor or NamedTensor is used, respectively. unflattened_size is the new shape of the unflattened dimension of the tensor and it can be a tuple of ints or a list of ints or tensorplay.Size for Tensor input; a NamedShape (tuple of (name, size) tuples) for NamedTensor input. Shape: Input: \((*, S_{\text{dim}}, *)\) , where \(S_{\text{dim}}\) is the size at dimension dim and \(*\) means any number of dimensions including none. Output: \((*, U_1, ..., U_n, *)\) , where \(U\) = unflattened_size and \(\prod_{i=1}^n U_i = S_{\text{dim}}\) . Parameters : dim ( Union [ int , str ] ) – Dimension to be unflattened unflattened_size ( Union [ tensorplay.Size , Tuple , List , NamedShape ] ) – New shape of the unflattened dimension Examples >>> input = tensorplay . randn ( 2 , 50 ) >>> # With tuple of ints >>> m = nn . Sequential ( >>> nn . Linear ( 50 , 50 ), >>> nn . Unflatten ( 1 , ( 2 , 5 , 5 )) >>> ) >>> output = m ( input ) >>> output . size () tensorplay.Size([2, 2, 5, 5]) >>> # With tensorplay.Size >>> m = nn . Sequential ( >>> nn . Linear ( 50 , 50 ), >>> nn . Unflatten ( 1 , tensorplay . Size ([ 2 , 5 , 5 ])) >>> ) >>> output = m ( input ) >>> output . size () tensorplay.Size([2, 2, 5, 5]) >>> # With namedshape (tuple of tuples) >>> input = tensorplay . randn ( 2 , 50 , names = ( "N" , "features" )) >>> unflatten = nn . Unflatten ( "features" , (( "C" , 2 ), ( "H" , 5 ), ( "W" , 5 ))) >>> output = unflatten ( input ) >>> output . size () tensorplay.Size([2, 2, 5, 5]) NamedShape alias of tuple [ tuple [ str , int ]] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str [source] Returns the extra representation of the module. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Fold URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.folding.Fold.html Fold class tensorplay.nn.modules.folding. Fold ( output_size , kernel_size , dilation = 1 , padding = 0 , stride = 1 ) [source] Combines an array of sliding local blocks into a large containing tensor (torch torch.nn.Fold ). Examples: >>> fold = nn . Fold ( output_size = ( 4 , 5 ), kernel_size = ( 2 , 2 )) >>> input = tp . randn ( 1 , 3 * 2 * 2 , 12 ) >>> output = fold ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Unfold URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.folding.Unfold.html Unfold class tensorplay.nn.modules.folding. Unfold ( kernel_size , dilation = 1 , padding = 0 , stride = 1 ) [source] Extracts sliding local blocks from a batched input tensor (torch torch.nn.Unfold ). Examples: >>> unfold = nn . Unfold ( kernel_size = ( 2 , 3 )) >>> input = tp . randn ( 2 , 5 , 3 , 4 ) >>> output = unfold ( input ) # (2, 5 * 2 * 3, 2 * 2) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # InstanceNorm1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.instancenorm.InstanceNorm1d.html InstanceNorm1d class tensorplay.nn.modules.instancenorm. InstanceNorm1d ( num_features : int , eps : float = 1e-05 , momentum : float = 0.1 , affine : bool = False , track_running_stats : bool = False , device = None , dtype = None ) [source] Applies Instance Normalization. This operation applies Instance Normalization over a 2D (unbatched) or 3D (batched) input as described in the paper Instance Normalization: The Missing Ingredient for Fast Stylization . \[y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta\] The mean and standard-deviation are calculated per-dimension separately for each object in a mini-batch. \(\gamma\) and \(\beta\) are learnable parameter vectors of size C (where C is the number of features or channels of the input) if affine is True . The variance is calculated via the biased estimator, equivalent to tensorplay.var(input, unbiased=False) . By default, this layer uses instance statistics computed from input data in both training and evaluation modes. If track_running_stats is set to True , during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default momentum of 0.1. Note This momentum argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is \(\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t\) , where \(\hat{x}\) is the estimated statistic and \(x_t\) is the new observed value. Note InstanceNorm1d and LayerNorm are very similar, but have some subtle differences. InstanceNorm1d is applied on each channel of channeled data like multidimensional time series, but LayerNorm is usually applied on entire sample and often in NLP tasks. Additionally, LayerNorm applies elementwise affine transform, while InstanceNorm1d usually don’t apply affine transform. Parameters : num_features – number of features or channels \(C\) of the input eps – a value added to the denominator for numerical stability. Default: 1e-5 momentum – the value used for the running_mean and running_var computation. Default: 0.1 affine – a boolean value that when set to True , this module has learnable affine parameters, initialized the same way as done for batch normalization. Default: False . track_running_stats – a boolean value that when set to True , this module tracks the running mean and variance, and when set to False , this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: False Shape: Input: \((N, C, L)\) or \((C, L)\) Output: \((N, C, L)\) or \((C, L)\) (same shape as input) Examples: >>> # Without Learnable Parameters >>> m = nn . InstanceNorm1d ( 100 ) >>> # With Learnable Parameters >>> m = nn . InstanceNorm1d ( 100 , affine = True ) >>> input = tensorplay . randn ( 20 , 100 , 40 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # InstanceNorm2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.instancenorm.InstanceNorm2d.html InstanceNorm2d class tensorplay.nn.modules.instancenorm. InstanceNorm2d ( num_features : int , eps : float = 1e-05 , momentum : float = 0.1 , affine : bool = False , track_running_stats : bool = False , device = None , dtype = None ) [source] Applies Instance Normalization. This operation applies Instance Normalization over a 4D input (a mini-batch of 2D inputs with additional channel dimension) as described in the paper Instance Normalization: The Missing Ingredient for Fast Stylization . \[y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta\] The mean and standard-deviation are calculated per-dimension separately for each object in a mini-batch. \(\gamma\) and \(\beta\) are learnable parameter vectors of size C (where C is the input size) if affine is True . The standard-deviation is calculated via the biased estimator, equivalent to tensorplay.var(input, unbiased=False) . By default, this layer uses instance statistics computed from input data in both training and evaluation modes. If track_running_stats is set to True , during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default momentum of 0.1. Note This momentum argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is \(\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t\) , where \(\hat{x}\) is the estimated statistic and \(x_t\) is the new observed value. Note InstanceNorm2d and LayerNorm are very similar, but have some subtle differences. InstanceNorm2d is applied on each channel of channeled data like RGB images, but LayerNorm is usually applied on entire sample and often in NLP tasks. Additionally, LayerNorm applies elementwise affine transform, while InstanceNorm2d usually don’t apply affine transform. Parameters : num_features – \(C\) from an expected input of size \((N, C, H, W)\) or \((C, H, W)\) eps – a value added to the denominator for numerical stability. Default: 1e-5 momentum – the value used for the running_mean and running_var computation. Default: 0.1 affine – a boolean value that when set to True , this module has learnable affine parameters, initialized the same way as done for batch normalization. Default: False . track_running_stats – a boolean value that when set to True , this module tracks the running mean and variance, and when set to False , this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: False Shape: Input: \((N, C, H, W)\) or \((C, H, W)\) Output: \((N, C, H, W)\) or \((C, H, W)\) (same shape as input) Examples: >>> # Without Learnable Parameters >>> m = nn . InstanceNorm2d ( 100 ) >>> # With Learnable Parameters >>> m = nn . InstanceNorm2d ( 100 , affine = True ) >>> input = tensorplay . randn ( 20 , 100 , 35 , 45 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # InstanceNorm3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.instancenorm.InstanceNorm3d.html InstanceNorm3d class tensorplay.nn.modules.instancenorm. InstanceNorm3d ( num_features : int , eps : float = 1e-05 , momentum : float = 0.1 , affine : bool = False , track_running_stats : bool = False , device = None , dtype = None ) [source] Applies Instance Normalization. This operation applies Instance Normalization over a 5D input (a mini-batch of 3D inputs with additional channel dimension) as described in the paper Instance Normalization: The Missing Ingredient for Fast Stylization . \[y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta\] The mean and standard-deviation are calculated per-dimension separately for each object in a mini-batch. \(\gamma\) and \(\beta\) are learnable parameter vectors of size C (where C is the input size) if affine is True . The standard-deviation is calculated via the biased estimator, equivalent to tensorplay.var(input, unbiased=False) . By default, this layer uses instance statistics computed from input data in both training and evaluation modes. If track_running_stats is set to True , during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default momentum of 0.1. Note This momentum argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is \(\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t\) , where \(\hat{x}\) is the estimated statistic and \(x_t\) is the new observed value. Note InstanceNorm3d and LayerNorm are very similar, but have some subtle differences. InstanceNorm3d is applied on each channel of channeled data like 3D models with RGB color, but LayerNorm is usually applied on entire sample and often in NLP tasks. Additionally, LayerNorm applies elementwise affine transform, while InstanceNorm3d usually don’t apply affine transform. Parameters : num_features – \(C\) from an expected input of size \((N, C, D, H, W)\) or \((C, D, H, W)\) eps – a value added to the denominator for numerical stability. Default: 1e-5 momentum – the value used for the running_mean and running_var computation. Default: 0.1 affine – a boolean value that when set to True , this module has learnable affine parameters, initialized the same way as done for batch normalization. Default: False . track_running_stats – a boolean value that when set to True , this module tracks the running mean and variance, and when set to False , this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: False Shape: Input: \((N, C, D, H, W)\) or \((C, D, H, W)\) Output: \((N, C, D, H, W)\) or \((C, D, H, W)\) (same shape as input) Examples: >>> # Without Learnable Parameters >>> m = nn . InstanceNorm3d ( 100 ) >>> # With Learnable Parameters >>> m = nn . InstanceNorm3d ( 100 , affine = True ) >>> input = tensorplay . randn ( 20 , 100 , 35 , 45 , 10 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # LazyInstanceNorm1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.instancenorm.LazyInstanceNorm1d.html LazyInstanceNorm1d class tensorplay.nn.modules.instancenorm. LazyInstanceNorm1d ( eps = 1e-05 , momentum = 0.1 , affine = True , track_running_stats = True , device = None , dtype = None ) [source] A tensorplay.nn.InstanceNorm1d module with lazy initialization of the num_features argument. The num_features argument of the InstanceNorm1d is inferred from the input.size(1) . The attributes that will be lazily initialized are weight , bias , running_mean and running_var . Check the tensorplay.nn.modules.lazy.LazyModuleMixin for further documentation on lazy modules and their limitations. Parameters : num_features – \(C\) from an expected input of size \((N, C, L)\) or \((C, L)\) eps – a value added to the denominator for numerical stability. Default: 1e-5 momentum – the value used for the running_mean and running_var computation. Default: 0.1 affine – a boolean value that when set to True , this module has learnable affine parameters, initialized the same way as done for batch normalization. Default: False . track_running_stats – a boolean value that when set to True , this module tracks the running mean and variance, and when set to False , this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: False Shape: Input: \((N, C, L)\) or \((C, L)\) Output: \((N, C, L)\) or \((C, L)\) (same shape as input) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module cls_to_become alias of InstanceNorm1d compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module has_uninitialized_params ( ) Check if a module has parameters that are not initialized. load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # LazyInstanceNorm2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.instancenorm.LazyInstanceNorm2d.html LazyInstanceNorm2d class tensorplay.nn.modules.instancenorm. LazyInstanceNorm2d ( eps = 1e-05 , momentum = 0.1 , affine = True , track_running_stats = True , device = None , dtype = None ) [source] A tensorplay.nn.InstanceNorm2d module with lazy initialization of the num_features argument. The num_features argument of the InstanceNorm2d is inferred from the input.size(1) . The attributes that will be lazily initialized are weight , bias , running_mean and running_var . Check the tensorplay.nn.modules.lazy.LazyModuleMixin for further documentation on lazy modules and their limitations. Parameters : num_features – \(C\) from an expected input of size \((N, C, H, W)\) or \((C, H, W)\) eps – a value added to the denominator for numerical stability. Default: 1e-5 momentum – the value used for the running_mean and running_var computation. Default: 0.1 affine – a boolean value that when set to True , this module has learnable affine parameters, initialized the same way as done for batch normalization. Default: False . track_running_stats – a boolean value that when set to True , this module tracks the running mean and variance, and when set to False , this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: False Shape: Input: \((N, C, H, W)\) or \((C, H, W)\) Output: \((N, C, H, W)\) or \((C, H, W)\) (same shape as input) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module cls_to_become alias of InstanceNorm2d compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module has_uninitialized_params ( ) Check if a module has parameters that are not initialized. load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # LazyInstanceNorm3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.instancenorm.LazyInstanceNorm3d.html LazyInstanceNorm3d class tensorplay.nn.modules.instancenorm. LazyInstanceNorm3d ( eps = 1e-05 , momentum = 0.1 , affine = True , track_running_stats = True , device = None , dtype = None ) [source] A tensorplay.nn.InstanceNorm3d module with lazy initialization of the num_features argument. The num_features argument of the InstanceNorm3d is inferred from the input.size(1) . The attributes that will be lazily initialized are weight , bias , running_mean and running_var . Check the tensorplay.nn.modules.lazy.LazyModuleMixin for further documentation on lazy modules and their limitations. Parameters : num_features – \(C\) from an expected input of size \((N, C, D, H, W)\) or \((C, D, H, W)\) eps – a value added to the denominator for numerical stability. Default: 1e-5 momentum – the value used for the running_mean and running_var computation. Default: 0.1 affine – a boolean value that when set to True , this module has learnable affine parameters, initialized the same way as done for batch normalization. Default: False . track_running_stats – a boolean value that when set to True , this module tracks the running mean and variance, and when set to False , this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: False Shape: Input: \((N, C, D, H, W)\) or \((C, D, H, W)\) Output: \((N, C, D, H, W)\) or \((C, D, H, W)\) (same shape as input) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module cls_to_become alias of InstanceNorm3d compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module has_uninitialized_params ( ) Check if a module has parameters that are not initialized. load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # LazyLinear URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.lazy.LazyLinear.html LazyLinear class tensorplay.nn.modules.lazy. LazyLinear ( out_features : int , bias : bool = True ) [source] A tensorplay.nn.Linear module where in_features is inferred. In this module, the weight and bias are of tensorplay.nn.UninitializedParameter class. They will be initialized after the first call to forward is done and the module will become a regular tensorplay.nn.Linear module. The in_features argument of the Linear is inferred from the input.shape[-1] . Parameters : out_features – size of each output sample bias – If set to False , the layer will not learn an additive bias. Default: True weight the learnable weights of the module of shape \((\text{out\_features}, \text{in\_features})\) . Type : tensorplay.nn.parameter.UninitializedParameter bias the learnable bias of the module of shape \((\text{out\_features})\) . add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module cls_to_become alias of Linear compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module has_uninitialized_params ( ) Check if a module has parameters that are not initialized. initialize_parameters ( input ) → None [source] Infers in_features based on input and initializes parameters. load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module reset_parameters ( ) → None [source] Resets parameters based on their initialization used in __init__ . set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # LazyModuleMixin URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.lazy.LazyModuleMixin.html LazyModuleMixin class tensorplay.nn.modules.lazy. LazyModuleMixin ( * args , ** kwargs ) [source] A mixin for modules that lazily initialize parameters, also known as “lazy modules”. Modules that lazily initialize parameters, or “lazy modules”, derive the shapes of their parameters from the first input(s) to their forward method. Until that first forward they contain tensorplay.nn.UninitializedParameter s that should not be accessed or used, and afterward they contain regular tensorplay.nn.Parameter s. Lazy modules are convenient since they don’t require computing some module arguments, like the in_features argument of a typical tensorplay.nn.Linear . After construction, networks with lazy modules should first be converted to the desired dtype and placed on the expected device. This is because lazy modules only perform shape inference so the usual dtype and device placement behavior applies. The lazy modules should then perform “dry runs” to initialize all the components in the module. These “dry runs” send inputs of the correct size, dtype, and device through the network and to each one of its lazy modules. After this the network can be used as usual. >>> # xdoctest: +SKIP >>> class LazyMLP ( tensorplay . nn . Module ): ... def __init__ ( self ) -> None : ... super () . __init__ () ... self . fc1 = tensorplay . nn . LazyLinear ( 10 ) ... self . relu1 = tensorplay . nn . ReLU () ... self . fc2 = tensorplay . nn . LazyLinear ( 1 ) ... self . relu2 = tensorplay . nn . ReLU () ... ... def forward ( self , input ): ... x = self . relu1 ( self . fc1 ( input )) ... y = self . relu2 ( self . fc2 ( x )) ... return y >>> # constructs a network with lazy modules >>> lazy_mlp = LazyMLP () >>> # transforms the network's device and dtype >>> # NOTE: these transforms can and should be applied after construction and before any 'dry runs' >>> lazy_mlp = lazy_mlp . cuda () >>> lazy_mlp LazyMLP( (fc1): LazyLinear(in_features=0, out_features=10, bias=True) (relu1): ReLU() (fc2): LazyLinear(in_features=0, out_features=1, bias=True) (relu2): ReLU() ) >>> # performs a dry run to initialize the network's lazy modules >>> lazy_mlp ( tensorplay . ones ( 10 , 10 ) . cuda ()) >>> # after initialization, LazyLinear modules become regular Linear modules >>> lazy_mlp LazyMLP( (fc1): Linear(in_features=10, out_features=10, bias=True) (relu1): ReLU() (fc2): Linear(in_features=10, out_features=1, bias=True) (relu2): ReLU() ) >>> # attaches an optimizer, since parameters can now be used as usual >>> optim = tensorplay . optim . SGD ( lazy_mlp . parameters (), lr = 0.01 ) A final caveat when using lazy modules is that the order of initialization of a network’s parameters may change, since the lazy modules are always initialized after other modules. For example, if the LazyMLP class defined above had a tensorplay.nn.LazyLinear module first and then a regular tensorplay.nn.Linear second, the second module would be initialized on construction and the first module would be initialized during the first dry run. This can cause the parameters of a network using lazy modules to be initialized differently than the parameters of a network without lazy modules as the order of parameter initializations, which often depends on a stateful random number generator, is different. Check the randomness documentation for more details. Lazy modules can be serialized with a state dict like other modules. For example: >>> lazy_mlp = LazyMLP () >>> # The state dict shows the uninitialized parameters >>> lazy_mlp . state_dict () OrderedDict({'fc1.weight': , 'fc1.bias': , 'fc2.weight': , 'fc2.bias': }) Lazy modules can load regular tensorplay.nn.Parameter s (i.e. you can serialize/deserialize initialized LazyModules and they will remain initialized) >>> full_mlp = LazyMLP () >>> # Dry run to initialize another module >>> full_mlp . forward ( tensorplay . ones ( 10 , 1 )) >>> # Load an initialized state into a lazy module >>> lazy_mlp . load_state_dict ( full_mlp . state_dict ()) >>> # The state dict now holds valid values >>> lazy_mlp . state_dict () OrderedDict([('fc1.weight', tensor([[-0.3837], [ 0.0907], [ 0.6708], [-0.5223], [-0.9028], [ 0.2851], [-0.4537], [ 0.6813], [ 0.5766], [-0.8678]])), ('fc1.bias', tensor([-1.8832e+25, 4.5636e-41, -1.8832e+25, 4.5636e-41, -6.1598e-30, 4.5637e-41, -1.8788e+22, 4.5636e-41, -2.0042e-31, 4.5637e-41])), ('fc2.weight', tensor([[ 0.1320, 0.2938, 0.0679, 0.2793, 0.1088, -0.1795, -0.2301, 0.2807, 0.2479, 0.1091]])), ('fc2.bias', tensor([0.0019]))]) Note, however, that the loaded parameters will not be replaced when doing a “dry run” if they are initialized when the state is loaded. This prevents using initialized modules in different contexts. has_uninitialized_params ( ) [source] Check if a module has parameters that are not initialized. initialize_parameters ( * args , ** kwargs ) [source] Initialize parameters according to the input batch properties. This adds an interface to isolate parameter initialization from the forward pass when doing parameter shape inference. # Bilinear URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.linear.Bilinear.html Bilinear class tensorplay.nn.modules.linear. Bilinear ( in1_features : int , in2_features : int , out_features : int , bias : bool = True , device = None , dtype = None ) [source] Applies a bilinear transformation to the incoming data: \(y = x_1^T A x_2 + b\) . Parameters : in1_features – size of each first input sample, must be > 0 in2_features – size of each second input sample, must be > 0 out_features – size of each output sample, must be > 0 bias – If set to False , the layer will not learn an additive bias. Default: True Shape: Input1: \((*, H_\text{in1})\) where \(H_\text{in1}=\text{in1\_features}\) and \(*\) means any number of additional dimensions including none. All but the last dimension of the inputs should be the same. Input2: \((*, H_\text{in2})\) where \(H_\text{in2}=\text{in2\_features}\) . Output: \((*, H_\text{out})\) where \(H_\text{out}=\text{out\_features}\) and all but the last dimension are the same shape as the input. weight the learnable weights of the module of shape \((\text{out\_features}, \text{in1\_features}, \text{in2\_features})\) . The values are initialized from \(\mathcal{U}(-\sqrt{k}, \sqrt{k})\) , where \(k = \frac{1}{\text{in1\_features}}\) Type : tensorplay._C.TensorBase bias the learnable bias of the module of shape \((\text{out\_features})\) . If bias is True , the values are initialized from \(\mathcal{U}(-\sqrt{k}, \sqrt{k})\) , where \(k = \frac{1}{\text{in1\_features}}\) Examples: >>> m = tp . nn . Bilinear ( 20 , 30 , 40 ) >>> input1 = tp . randn ( 128 , 20 ) >>> input2 = tp . randn ( 128 , 30 ) >>> output = m ( input1 , input2 ) >>> print ( output . size ()) tensorplay.Size([128, 40]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str [source] Return the extra representation of the module. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input1 : TensorBase , input2 : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module reset_parameters ( ) → None [source] Resets parameters based on their initialization used in __init__ . set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Identity URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.linear.Identity.html Identity class tensorplay.nn.modules.linear. Identity ( * args : Any , ** kwargs : Any ) [source] A placeholder identity operator that is argument-insensitive. Parameters : args – any argument (unused) kwargs – any keyword argument (unused) Shape: Input: \((*)\) , where \(*\) means any number of dimensions. Output: \((*)\) , same shape as the input. Examples: >>> m = tp . nn . Identity ( 54 , unused_argument1 = 0.1 , unused_argument2 = False ) >>> input = tp . randn ( 128 , 20 ) >>> output = m ( input ) >>> print ( output . size ()) tensorplay.Size([128, 20]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Linear URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.linear.Linear.html Linear class tensorplay.nn.modules.linear. Linear ( in_features : int , out_features : int , bias : bool = True , device = None , dtype = None ) [source] Applies an affine linear transformation to the incoming data: \(y = xA^T + b\) . Parameters : in_features – size of each input sample out_features – size of each output sample bias – If set to False , the layer will not learn an additive bias. Default: True Shape: Input: \((*, H_\text{in})\) where \(*\) means any number of dimensions including none and \(H_\text{in} = \text{in\_features}\) . Output: \((*, H_\text{out})\) where all but the last dimension are the same shape as the input and \(H_\text{out} = \text{out\_features}\) . weight the learnable weights of the module of shape \((\text{out\_features}, \text{in\_features})\) . The values are initialized from \(\mathcal{U}(-\sqrt{k}, \sqrt{k})\) , where \(k = \frac{1}{\text{in\_features}}\) Type : tensorplay._C.TensorBase bias the learnable bias of the module of shape \((\text{out\_features})\) . If bias is True , the values are initialized from \(\mathcal{U}(-\sqrt{k}, \sqrt{k})\) where \(k = \frac{1}{\text{in\_features}}\) Examples: >>> m = tp . nn . Linear ( 20 , 30 ) >>> input = tp . randn ( 128 , 20 ) >>> output = m ( input ) >>> print ( output . size ()) tensorplay.Size([128, 30]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str [source] Return the extra representation of the module. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module reset_parameters ( ) → None [source] Resets parameters based on their initialization used in __init__ . set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # BCELoss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.loss.BCELoss.html BCELoss class tensorplay.nn.modules.loss. BCELoss ( weight : TensorBase | None = None , size_average = None , reduce = None , reduction : str = 'mean' ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # BCEWithLogitsLoss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.loss.BCEWithLogitsLoss.html BCEWithLogitsLoss class tensorplay.nn.modules.loss. BCEWithLogitsLoss ( weight : TensorBase | None = None , size_average = None , reduce = None , reduction : str = 'mean' , pos_weight : TensorBase | None = None ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # CosineEmbeddingLoss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.loss.CosineEmbeddingLoss.html CosineEmbeddingLoss class tensorplay.nn.modules.loss. CosineEmbeddingLoss ( margin = 0.0 , size_average = None , reduce = None , reduction : str = 'mean' ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # CrossEntropyLoss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.loss.CrossEntropyLoss.html CrossEntropyLoss class tensorplay.nn.modules.loss. CrossEntropyLoss ( weight : TensorBase | None = None , size_average = None , ignore_index : int = -100 , reduce = None , reduction : str = 'mean' , label_smoothing : float = 0.0 ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # CTCLoss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.loss.CTCLoss.html CTCLoss class tensorplay.nn.modules.loss. CTCLoss ( blank : int = 0 , reduction : str = 'mean' , zero_infinity : bool = False ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # GaussianNLLLoss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.loss.GaussianNLLLoss.html GaussianNLLLoss class tensorplay.nn.modules.loss. GaussianNLLLoss ( * , full : bool = False , eps : float = 1e-06 , reduction : str = 'mean' ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # HingeEmbeddingLoss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.loss.HingeEmbeddingLoss.html HingeEmbeddingLoss class tensorplay.nn.modules.loss. HingeEmbeddingLoss ( margin : float = 1.0 , size_average = None , reduce = None , reduction : str = 'mean' ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # HuberLoss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.loss.HuberLoss.html HuberLoss class tensorplay.nn.modules.loss. HuberLoss ( reduction : str = 'mean' , delta : float = 1.0 ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # KLDivLoss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.loss.KLDivLoss.html KLDivLoss class tensorplay.nn.modules.loss. KLDivLoss ( reduction : str = 'mean' , log_target : bool = False ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # L1Loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.loss.L1Loss.html L1Loss class tensorplay.nn.modules.loss. L1Loss ( reduction : str = 'mean' ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # MarginRankingLoss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.loss.MarginRankingLoss.html MarginRankingLoss class tensorplay.nn.modules.loss. MarginRankingLoss ( margin : float = 0.0 , size_average = None , reduce = None , reduction : str = 'mean' ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # MSELoss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.loss.MSELoss.html MSELoss class tensorplay.nn.modules.loss. MSELoss ( reduction : str = 'mean' ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # MultiLabelMarginLoss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.loss.MultiLabelMarginLoss.html MultiLabelMarginLoss class tensorplay.nn.modules.loss. MultiLabelMarginLoss ( size_average = None , reduce = None , reduction : str = 'mean' ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # MultiLabelSoftMarginLoss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.loss.MultiLabelSoftMarginLoss.html MultiLabelSoftMarginLoss class tensorplay.nn.modules.loss. MultiLabelSoftMarginLoss ( weight : TensorBase | None = None , size_average = None , reduce = None , reduction : str = 'mean' ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # MultiMarginLoss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.loss.MultiMarginLoss.html MultiMarginLoss class tensorplay.nn.modules.loss. MultiMarginLoss ( p : int = 1 , margin : float = 1.0 , weight : TensorBase | None = None , size_average = None , reduce = None , reduction : str = 'mean' ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # NLLLoss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.loss.NLLLoss.html NLLLoss class tensorplay.nn.modules.loss. NLLLoss ( weight : TensorBase | None = None , size_average = None , ignore_index : int = -100 , reduce = None , reduction : str = 'mean' ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # PoissonNLLLoss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.loss.PoissonNLLLoss.html PoissonNLLLoss class tensorplay.nn.modules.loss. PoissonNLLLoss ( * , log_input : bool = True , full : bool = False , eps : float = 1e-08 , reduction : str = 'mean' ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # SmoothL1Loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.loss.SmoothL1Loss.html SmoothL1Loss class tensorplay.nn.modules.loss. SmoothL1Loss ( reduction : str = 'mean' , beta : float = 1.0 ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # SoftMarginLoss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.loss.SoftMarginLoss.html SoftMarginLoss class tensorplay.nn.modules.loss. SoftMarginLoss ( size_average = None , reduce = None , reduction : str = 'mean' ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # TripletMarginLoss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.loss.TripletMarginLoss.html TripletMarginLoss class tensorplay.nn.modules.loss. TripletMarginLoss ( margin = 1.0 , p = 2.0 , eps = 1e-06 , swap = False , size_average = None , reduce = None , reduction : str = 'mean' ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # TripletMarginWithDistanceLoss URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.loss.TripletMarginWithDistanceLoss.html TripletMarginWithDistanceLoss class tensorplay.nn.modules.loss. TripletMarginWithDistanceLoss ( * , distance_function = None , margin : float = 1.0 , swap : bool = False , reduction : str = 'mean' ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Module URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.module.Module.html Module class tensorplay.nn.modules.module. Module ( * args , ** kwargs ) [source] Base class for all neural network modules. Your models should also subclass this class. Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes: import tensorplay.nn as nn import tensorplay.nn.functional as F class Model ( nn . Module ): def __init__ ( self ) -> None : super () . __init__ () self . conv1 = nn . Conv2d ( 1 , 20 , 5 ) self . conv2 = nn . Conv2d ( 20 , 20 , 5 ) def forward ( self , x ): x = F . relu ( self . conv1 ( x )) return F . relu ( self . conv2 ( x )) Submodules assigned in this way will be registered, and will also have their parameters converted when you call to() , etc. Note As per the example above, an __init__() call to the parent class must be made before assignment on the child. Variables : training ( bool ) – Boolean represents whether this module is in training or evaluation mode. add_module ( name : str , module : Module | None ) → None [source] Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self [source] Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self [source] Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] [source] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] [source] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) [source] Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self [source] Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self [source] Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self [source] Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self [source] Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str [source] Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self [source] Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( * input : Any ) → None Define the computation performed at every call. Should be overridden by all subclasses. Note Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them. get_buffer ( target : str ) → TensorBase [source] Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any [source] Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter [source] Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module [source] Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self [source] Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) [source] Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] [source] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] [source] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] [source] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) [source] Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] [source] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] [source] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle [source] Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None [source] Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle [source] Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle [source] Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle [source] Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle [source] Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) [source] Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) [source] Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None [source] Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None [source] Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) [source] Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) [source] Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self [source] Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None [source] Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None [source] Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self [source] See tensorplay.Tensor.share_memory_() . state_dict ( * , destination : T_destination , prefix : str = '' , keep_vars : bool = False ) → T_destination [source] state_dict ( * , prefix : str = '' , keep_vars : bool = False ) → dict [ str , Any ] Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( device : str | Device | int | None = ... , dtype : DType | None = ... , non_blocking : bool = ... ) → Self [source] to ( dtype : DType , non_blocking : bool = ... ) → Self to ( tensor : TensorBase , non_blocking : bool = ... ) → Self Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) [source] to ( dtype , non_blocking = False ) [source] to ( tensor , non_blocking = False ) [source] to ( memory_format = tensorplay.channels_last ) [source] Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self [source] Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self [source] Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self [source] Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None [source] Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # tensorplay.nn.modules.module.register_module_backward_hook URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.module.register_module_backward_hook.html tensorplay.nn.modules.module.register_module_backward_hook tensorplay.nn.modules.module. register_module_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle [source] Register a backward hook common to all the modules. This function is deprecated in favor of tensorplay.nn.modules.module.register_module_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle # tensorplay.nn.modules.module.register_module_buffer_registration_hook URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.module.register_module_buffer_registration_hook.html tensorplay.nn.modules.module.register_module_buffer_registration_hook tensorplay.nn.modules.module. register_module_buffer_registration_hook ( hook : Callable [ [ ... ] , None ] ) → RemovableHandle [source] Register a buffer registration hook common to all modules. Warning This adds global state to the nn.Module module The hook will be called every time register_buffer() is invoked. It should have the following signature: hook ( module , name , buffer ) -> None or new buffer The hook can modify the input or return a single modified value in the hook. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle # tensorplay.nn.modules.module.register_module_forward_hook URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.module.register_module_forward_hook.html tensorplay.nn.modules.module.register_module_forward_hook tensorplay.nn.modules.module. register_module_forward_hook ( hook : Callable [ [ ... ] , None ] , * , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle [source] Register a global forward hook for all the modules. Warning This adds global state to the nn.module module and it is only intended for debugging/profiling purposes. The hook will be called every time after forward() has computed an output. It should have the following signature: hook ( module , input , output ) -> None or modified output The input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . You can optionally modify the output of the module by returning a new value that will replace the output from the forward() function. Parameters : hook ( Callable ) – The user defined hook to be registered. always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle This hook will be executed before specific module hooks registered with register_forward_hook . # tensorplay.nn.modules.module.register_module_forward_pre_hook URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.module.register_module_forward_pre_hook.html tensorplay.nn.modules.module.register_module_forward_pre_hook tensorplay.nn.modules.module. register_module_forward_pre_hook ( hook : Callable [ [ ... ] , None ] ) → RemovableHandle [source] Register a forward pre-hook common to all modules. Warning This adds global state to the nn.module module and it is only intended for debugging/profiling purposes. The hook will be called every time before forward() is invoked. It should have the following signature: hook ( module , input ) -> None or modified input The input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned(unless that value is already a tuple). This hook has precedence over the specific module hooks registered with register_forward_pre_hook . Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle # tensorplay.nn.modules.module.register_module_full_backward_hook URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.module.register_module_full_backward_hook.html tensorplay.nn.modules.module.register_module_full_backward_hook tensorplay.nn.modules.module. register_module_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle [source] Register a backward hook common to all the modules. Warning This adds global state to the nn.module module and it is only intended for debugging/profiling purposes. Hooks registered using this function behave in the same way as those registered by tensorplay.nn.Module.register_full_backward_hook() . Refer to its documentation for more details. Hooks registered using this function will be called before hooks registered using tensorplay.nn.Module.register_full_backward_hook() . Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle # tensorplay.nn.modules.module.register_module_full_backward_pre_hook URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.module.register_module_full_backward_pre_hook.html tensorplay.nn.modules.module.register_module_full_backward_pre_hook tensorplay.nn.modules.module. register_module_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle [source] Register a backward pre-hook common to all the modules. Warning This adds global state to the nn.module module and it is only intended for debugging/profiling purposes. Hooks registered using this function behave in the same way as those registered by tensorplay.nn.Module.register_full_backward_pre_hook() . Refer to its documentation for more details. Hooks registered using this function will be called before hooks registered using tensorplay.nn.Module.register_full_backward_pre_hook() . Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle # tensorplay.nn.modules.module.register_module_module_registration_hook URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.module.register_module_module_registration_hook.html tensorplay.nn.modules.module.register_module_module_registration_hook tensorplay.nn.modules.module. register_module_module_registration_hook ( hook : Callable [ [ ... ] , None ] ) → RemovableHandle [source] Register a module registration hook common to all modules. Warning This adds global state to the nn.Module module The hook will be called every time register_module() is invoked. It should have the following signature: hook ( module , name , submodule ) -> None or new submodule The hook can modify the input or return a single modified value in the hook. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle # tensorplay.nn.modules.module.register_module_parameter_registration_hook URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.module.register_module_parameter_registration_hook.html tensorplay.nn.modules.module.register_module_parameter_registration_hook tensorplay.nn.modules.module. register_module_parameter_registration_hook ( hook : Callable [ [ ... ] , None ] ) → RemovableHandle [source] Register a parameter registration hook common to all modules. Warning This adds global state to the nn.Module module The hook will be called every time register_parameter() is invoked. It should have the following signature: hook ( module , name , param ) -> None or new parameter The hook can modify the input or return a single modified value in the hook. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle # MultiheadAttention URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.multihead_attention.MultiheadAttention.html MultiheadAttention class tensorplay.nn.modules.multihead_attention. MultiheadAttention ( embed_dim : int , num_heads : int , dropout : float = 0.0 , bias : bool = True , add_bias_kv : bool = False , add_zero_attn : bool = False , kdim : int | None = None , vdim : int | None = None ) [source] Allows the model to jointly attend to information from different representation subspaces, as described in the paper Attention Is All You Need . Args mirror torch.nn.MultiheadAttention; batch_first=True is required by the torchvision transformer models and is supported here. add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # GroupNorm URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.normalization.GroupNorm.html GroupNorm class tensorplay.nn.modules.normalization. GroupNorm ( num_groups : int , num_channels : int , eps : float = 1e-05 , affine : bool = True , device = None , dtype = None ) [source] Applies Group Normalization over a mini-batch of inputs. This layer implements the operation as described in the paper Group Normalization \[y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta\] The input channels are separated into num_groups groups, each containing num_channels / num_groups channels. num_channels must be divisible by num_groups . The mean and standard-deviation are calculated separately over each group. \(\gamma\) and \(\beta\) are learnable per-channel affine transform parameter vectors of size num_channels if affine is True . The variance is calculated via the biased estimator, equivalent to tensorplay.var(input, unbiased=False) . This layer uses statistics computed from input data in both training and evaluation modes. Parameters : num_groups ( int ) – number of groups to separate the channels into num_channels ( int ) – number of channels expected in input eps – a value added to the denominator for numerical stability. Default: 1e-5 affine – a boolean value that when set to True , this module has learnable per-channel affine parameters initialized to ones (for weights) and zeros (for biases). Default: True . Shape: Input: \((N, C, *)\) where \(C=\text{num\_channels}\) Output: \((N, C, *)\) (same shape as input) Examples: >>> input = tensorplay . randn ( 20 , 6 , 10 , 10 ) >>> # Separate 6 channels into 3 groups >>> m = nn . GroupNorm ( 3 , 6 ) >>> # Separate 6 channels into 6 groups (equivalent with InstanceNorm) >>> m = nn . GroupNorm ( 6 , 6 ) >>> # Put all 6 channels into a single group (equivalent with LayerNorm) >>> m = nn . GroupNorm ( 1 , 6 ) >>> # Activating the module >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # LayerNorm URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.normalization.LayerNorm.html LayerNorm class tensorplay.nn.modules.normalization. LayerNorm ( normalized_shape : int | list [ int ] | Size , eps : float = 1e-05 , elementwise_affine : bool = True , bias : bool = True , device = None , dtype = None ) [source] Applies Layer Normalization over a mini-batch of inputs. This layer implements the operation as described in the paper Layer Normalization \[y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta\] The mean and standard-deviation are calculated over the last D dimensions, where D is the dimension of normalized_shape . For example, if normalized_shape is (3, 5) (a 2-dimensional shape), the mean and standard-deviation are computed over the last 2 dimensions of the input (i.e. input.mean((-2, -1)) ). \(\gamma\) and \(\beta\) are learnable affine transform parameters of normalized_shape if elementwise_affine is True . The variance is calculated via the biased estimator, equivalent to tensorplay.var(input, unbiased=False) . Note Unlike Batch Normalization and Instance Normalization, which applies scalar scale and bias for each entire channel/plane with the affine option, Layer Normalization applies per-element scale and bias with elementwise_affine . This layer uses statistics computed from input data in both training and evaluation modes. Parameters : normalized_shape ( int or list or tensorplay.Size ) – input shape from an expected input of size \[[* \times \text{normalized\_shape}[0] \times \text{normalized\_shape}[1] \times \ldots \times \text{normalized\_shape}[-1]]\] If a single integer is used, it is treated as a singleton list, and this module will normalize over the last dimension which is expected to be of that specific size. eps – a value added to the denominator for numerical stability. Default: 1e-5 elementwise_affine – a boolean value that when set to True , this module has learnable per-element affine parameters initialized to ones (for weights) and zeros (for biases). Default: True . bias – If set to False , the layer will not learn an additive bias (only relevant if elementwise_affine is True ). Default: True . weight the learnable weights of the module of shape \(\text{normalized\_shape}\) when elementwise_affine is set to True . The values are initialized to 1. bias the learnable bias of the module of shape \(\text{normalized\_shape}\) when elementwise_affine is set to True . The values are initialized to 0. Shape: Input: \((N, *)\) Output: \((N, *)\) (same shape as input) Examples: >>> # NLP Example >>> batch , sentence_length , embedding_dim = 20 , 5 , 10 >>> embedding = tensorplay . randn ( batch , sentence_length , embedding_dim ) >>> layer_norm = nn . LayerNorm ( embedding_dim ) >>> # Activate module >>> layer_norm ( embedding ) >>> >>> # Image Example >>> N , C , H , W = 20 , 5 , 10 , 10 >>> input = tensorplay . randn ( N , C , H , W ) >>> # Normalize over the last three dimensions (i.e. the channel and spatial dimensions) >>> # as shown in the image below >>> layer_norm = nn . LayerNorm ([ C , H , W ]) >>> output = layer_norm ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # LocalResponseNorm URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.normalization.LocalResponseNorm.html LocalResponseNorm class tensorplay.nn.modules.normalization. LocalResponseNorm ( size : int , alpha : float = 0.0001 , beta : float = 0.75 , k : float = 1.0 ) [source] Applies local response normalization over an input signal. The input signal is composed of several input planes, where channels occupy the second dimension. Applies normalization across channels. \[b_{c} = a_{c}\left(k + \frac{\alpha}{n} \sum_{c'=\max(0, c-n/2)}^{\min(N-1,c+n/2)}a_{c'}^2\right)^{-\beta}\] Parameters : size – amount of neighbouring channels used for normalization alpha – multiplicative factor. Default: 0.0001 beta – exponent. Default: 0.75 k – additive factor. Default: 1 Shape: Input: \((N, C, *)\) Output: \((N, C, *)\) (same shape as input) Examples: >>> lrn = nn . LocalResponseNorm ( 2 ) >>> signal_2d = tensorplay . randn ( 32 , 5 , 24 , 24 ) >>> signal_4d = tensorplay . randn ( 16 , 5 , 7 , 7 , 7 , 7 ) >>> output_2d = lrn ( signal_2d ) >>> output_4d = lrn ( signal_4d ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) [source] Return the extra representation of the module. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # RMSNorm URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.normalization.RMSNorm.html RMSNorm class tensorplay.nn.modules.normalization. RMSNorm ( normalized_shape : int | list [ int ] | Size , eps : float | None = None , elementwise_affine : bool = True , device = None , dtype = None ) [source] Applies Root Mean Square Layer Normalization over a mini-batch of inputs. This layer implements the operation as described in the paper Root Mean Square Layer Normalization \[y_i = \frac{x_i}{\mathrm{RMS}(x)} * \gamma_i, \quad \text{where} \quad \text{RMS}(x) = \sqrt{\epsilon + \frac{1}{n} \sum_{i=1}^{n} x_i^2}\] The RMS is taken over the last D dimensions, where D is the dimension of normalized_shape . For example, if normalized_shape is (3, 5) (a 2-dimensional shape), the RMS is computed over the last 2 dimensions of the input. Parameters : normalized_shape ( int or list or tensorplay.Size ) – input shape from an expected input of size \[[* \times \text{normalized\_shape}[0] \times \text{normalized\_shape}[1] \times \ldots \times \text{normalized\_shape}[-1]]\] If a single integer is used, it is treated as a singleton list, and this module will normalize over the last dimension which is expected to be of that specific size. eps – a value added to the denominator for numerical stability. Default: tensorplay.finfo(x.dtype).eps elementwise_affine – a boolean value that when set to True , this module has learnable per-element affine parameters initialized to ones (for weights). Default: True . Shape: Input: \((N, *)\) Output: \((N, *)\) (same shape as input) Examples: >>> rms_norm = nn . RMSNorm ([ 2 , 3 ]) >>> input = tensorplay . randn ( 2 , 2 , 3 ) >>> rms_norm ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str [source] Return the extra representation of the module. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( x : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module reset_parameters ( ) → None [source] Resets parameters based on their initialization used in __init__. set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # CircularPad1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.padding.CircularPad1d.html CircularPad1d class tensorplay.nn.modules.padding. CircularPad1d ( padding : int | tuple [ int , int ] ) [source] Pads the input tensor using circular padding of the input boundary. Tensor values at the beginning of the dimension are used to pad the end, and values at the end are used to pad the beginning. If negative padding is applied then the ends of the tensor get removed. For N -dimensional padding, use torch.nn.functional.pad() . Parameters : padding ( int , tuple ) – the size of the padding. If it is int , uses the same padding in all boundaries. If a 2- tuple , uses ( \(\text{padding\_left}\) , \(\text{padding\_right}\) ) Note that padding size should be less than or equal to the corresponding input dimension. Shape: Input: \((C, W_{in})\) or \((N, C, W_{in})\) . Output: \((C, W_{out})\) or \((N, C, W_{out})\) , where \(W_{out} = W_{in} + \text{padding\_left} + \text{padding\_right}\) Examples: >>> # xdoctest: +IGNORE_WANT("not sure why xdoctest is choking on this") >>> m = nn . CircularPad1d ( 2 ) >>> input = torch . arange ( 8 , dtype = torch . float ) . reshape ( 1 , 2 , 4 ) >>> input tensor([[[0., 1., 2., 3.], [4., 5., 6., 7.]]]) >>> m ( input ) tensor([[[2., 3., 0., 1., 2., 3., 0., 1.], [6., 7., 4., 5., 6., 7., 4., 5.]]]) >>> # using different paddings for different sides >>> m = nn . CircularPad1d (( 3 , 1 )) >>> m ( input ) tensor([[[1., 2., 3., 0., 1., 2., 3., 0.], [5., 6., 7., 4., 5., 6., 7., 4.]]]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # CircularPad2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.padding.CircularPad2d.html CircularPad2d class tensorplay.nn.modules.padding. CircularPad2d ( padding : int | tuple [ int , int , int , int ] ) [source] Pads the input tensor using circular padding of the input boundary. Tensor values at the beginning of the dimension are used to pad the end, and values at the end are used to pad the beginning. If negative padding is applied then the ends of the tensor get removed. For N -dimensional padding, use torch.nn.functional.pad() . Parameters : padding ( int , tuple ) – the size of the padding. If it is int , uses the same padding in all boundaries. If a 4- tuple , uses ( \(\text{padding\_left}\) , \(\text{padding\_right}\) , \(\text{padding\_top}\) , \(\text{padding\_bottom}\) ) Note that padding size should be less than or equal to the corresponding input dimension. Shape: Input: \((N, C, H_{in}, W_{in})\) or \((C, H_{in}, W_{in})\) . Output: \((N, C, H_{out}, W_{out})\) or \((C, H_{out}, W_{out})\) , where \(H_{out} = H_{in} + \text{padding\_top} + \text{padding\_bottom}\) \(W_{out} = W_{in} + \text{padding\_left} + \text{padding\_right}\) Examples: >>> m = nn . CircularPad2d ( 2 ) >>> input = torch . arange ( 9 , dtype = torch . float ) . reshape ( 1 , 1 , 3 , 3 ) >>> input tensor([[[[0., 1., 2.], [3., 4., 5.], [6., 7., 8.]]]]) >>> m ( input ) tensor([[[[4., 5., 3., 4., 5., 3., 4.], [7., 8., 6., 7., 8., 6., 7.], [1., 2., 0., 1., 2., 0., 1.], [4., 5., 3., 4., 5., 3., 4.], [7., 8., 6., 7., 8., 6., 7.], [1., 2., 0., 1., 2., 0., 1.], [4., 5., 3., 4., 5., 3., 4.]]]]) >>> # using different paddings for different sides >>> m = nn . CircularPad2d (( 1 , 1 , 2 , 0 )) >>> m ( input ) tensor([[[[5., 3., 4., 5., 3.], [8., 6., 7., 8., 6.], [2., 0., 1., 2., 0.], [5., 3., 4., 5., 3.], [8., 6., 7., 8., 6.]]]]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # CircularPad3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.padding.CircularPad3d.html CircularPad3d class tensorplay.nn.modules.padding. CircularPad3d ( padding : int | tuple [ int , int , int , int , int , int ] ) [source] Pads the input tensor using circular padding of the input boundary. Tensor values at the beginning of the dimension are used to pad the end, and values at the end are used to pad the beginning. If negative padding is applied then the ends of the tensor get removed. For N -dimensional padding, use torch.nn.functional.pad() . Parameters : padding ( int , tuple ) – the size of the padding. If it is int , uses the same padding in all boundaries. If a 6- tuple , uses ( \(\text{padding\_left}\) , \(\text{padding\_right}\) , \(\text{padding\_top}\) , \(\text{padding\_bottom}\) , \(\text{padding\_front}\) , \(\text{padding\_back}\) ) Note that padding size should be less than or equal to the corresponding input dimension. Shape: Input: \((N, C, D_{in}, H_{in}, W_{in})\) or \((C, D_{in}, H_{in}, W_{in})\) . Output: \((N, C, D_{out}, H_{out}, W_{out})\) or \((C, D_{out}, H_{out}, W_{out})\) , where \(D_{out} = D_{in} + \text{padding\_front} + \text{padding\_back}\) \(H_{out} = H_{in} + \text{padding\_top} + \text{padding\_bottom}\) \(W_{out} = W_{in} + \text{padding\_left} + \text{padding\_right}\) Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> m = nn . CircularPad3d ( 3 ) >>> input = torch . randn ( 16 , 3 , 8 , 320 , 480 ) >>> output = m ( input ) >>> # using different paddings for different sides >>> m = nn . CircularPad3d (( 3 , 3 , 6 , 6 , 1 , 1 )) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # ConstantPad1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.padding.ConstantPad1d.html ConstantPad1d class tensorplay.nn.modules.padding. ConstantPad1d ( padding : int | tuple [ int , int ] , value : float ) [source] Pads the input tensor boundaries with a constant value. For N -dimensional padding, use torch.nn.functional.pad() . Parameters : padding ( int , tuple ) – the size of the padding. If it is int , uses the same padding in both boundaries. If a 2- tuple , uses ( \(\text{padding\_left}\) , \(\text{padding\_right}\) ) Shape: Input: \((C, W_{in})\) or \((N, C, W_{in})\) . Output: \((C, W_{out})\) or \((N, C, W_{out})\) , where \(W_{out} = W_{in} + \text{padding\_left} + \text{padding\_right}\) Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> m = nn . ConstantPad1d ( 2 , 3.5 ) >>> input = torch . randn ( 1 , 2 , 4 ) >>> input tensor([[[-1.0491, -0.7152, -0.0749, 0.8530], [-1.3287, 1.8966, 0.1466, -0.2771]]]) >>> m ( input ) tensor([[[ 3.5000, 3.5000, -1.0491, -0.7152, -0.0749, 0.8530, 3.5000, 3.5000], [ 3.5000, 3.5000, -1.3287, 1.8966, 0.1466, -0.2771, 3.5000, 3.5000]]]) >>> m = nn . ConstantPad1d ( 2 , 3.5 ) >>> input = torch . randn ( 1 , 2 , 3 ) >>> input tensor([[[ 1.6616, 1.4523, -1.1255], [-3.6372, 0.1182, -1.8652]]]) >>> m ( input ) tensor([[[ 3.5000, 3.5000, 1.6616, 1.4523, -1.1255, 3.5000, 3.5000], [ 3.5000, 3.5000, -3.6372, 0.1182, -1.8652, 3.5000, 3.5000]]]) >>> # using different paddings for different sides >>> m = nn . ConstantPad1d (( 3 , 1 ), 3.5 ) >>> m ( input ) tensor([[[ 3.5000, 3.5000, 3.5000, 1.6616, 1.4523, -1.1255, 3.5000], [ 3.5000, 3.5000, 3.5000, -3.6372, 0.1182, -1.8652, 3.5000]]]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # ConstantPad2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.padding.ConstantPad2d.html ConstantPad2d class tensorplay.nn.modules.padding. ConstantPad2d ( padding : int | tuple [ int , int , int , int ] , value : float ) [source] Pads the input tensor boundaries with a constant value. For N -dimensional padding, use torch.nn.functional.pad() . Parameters : padding ( int , tuple ) – the size of the padding. If it is int , uses the same padding in all boundaries. If a 4- tuple , uses ( \(\text{padding\_left}\) , \(\text{padding\_right}\) , \(\text{padding\_top}\) , \(\text{padding\_bottom}\) ) Shape: Input: \((N, C, H_{in}, W_{in})\) or \((C, H_{in}, W_{in})\) . Output: \((N, C, H_{out}, W_{out})\) or \((C, H_{out}, W_{out})\) , where \(H_{out} = H_{in} + \text{padding\_top} + \text{padding\_bottom}\) \(W_{out} = W_{in} + \text{padding\_left} + \text{padding\_right}\) Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> m = nn . ConstantPad2d ( 2 , 3.5 ) >>> input = torch . randn ( 1 , 2 , 2 ) >>> input tensor([[[ 1.6585, 0.4320], [-0.8701, -0.4649]]]) >>> m ( input ) tensor([[[ 3.5000, 3.5000, 3.5000, 3.5000, 3.5000, 3.5000], [ 3.5000, 3.5000, 3.5000, 3.5000, 3.5000, 3.5000], [ 3.5000, 3.5000, 1.6585, 0.4320, 3.5000, 3.5000], [ 3.5000, 3.5000, -0.8701, -0.4649, 3.5000, 3.5000], [ 3.5000, 3.5000, 3.5000, 3.5000, 3.5000, 3.5000], [ 3.5000, 3.5000, 3.5000, 3.5000, 3.5000, 3.5000]]]) >>> # using different paddings for different sides >>> m = nn . ConstantPad2d (( 3 , 0 , 2 , 1 ), 3.5 ) >>> m ( input ) tensor([[[ 3.5000, 3.5000, 3.5000, 3.5000, 3.5000], [ 3.5000, 3.5000, 3.5000, 3.5000, 3.5000], [ 3.5000, 3.5000, 3.5000, 1.6585, 0.4320], [ 3.5000, 3.5000, 3.5000, -0.8701, -0.4649], [ 3.5000, 3.5000, 3.5000, 3.5000, 3.5000]]]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # ConstantPad3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.padding.ConstantPad3d.html ConstantPad3d class tensorplay.nn.modules.padding. ConstantPad3d ( padding : int | tuple [ int , int , int , int , int , int ] , value : float ) [source] Pads the input tensor boundaries with a constant value. For N -dimensional padding, use torch.nn.functional.pad() . Parameters : padding ( int , tuple ) – the size of the padding. If it is int , uses the same padding in all boundaries. If a 6- tuple , uses ( \(\text{padding\_left}\) , \(\text{padding\_right}\) , \(\text{padding\_top}\) , \(\text{padding\_bottom}\) , \(\text{padding\_front}\) , \(\text{padding\_back}\) ) Shape: Input: \((N, C, D_{in}, H_{in}, W_{in})\) or \((C, D_{in}, H_{in}, W_{in})\) . Output: \((N, C, D_{out}, H_{out}, W_{out})\) or \((C, D_{out}, H_{out}, W_{out})\) , where \(D_{out} = D_{in} + \text{padding\_front} + \text{padding\_back}\) \(H_{out} = H_{in} + \text{padding\_top} + \text{padding\_bottom}\) \(W_{out} = W_{in} + \text{padding\_left} + \text{padding\_right}\) Examples: >>> m = nn . ConstantPad3d ( 3 , 3.5 ) >>> input = torch . randn ( 16 , 3 , 10 , 20 , 30 ) >>> output = m ( input ) >>> # using different paddings for different sides >>> m = nn . ConstantPad3d (( 3 , 3 , 6 , 6 , 0 , 1 ), 3.5 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # ReflectionPad1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.padding.ReflectionPad1d.html ReflectionPad1d class tensorplay.nn.modules.padding. ReflectionPad1d ( padding : int | tuple [ int , int ] ) [source] Pads the input tensor using the reflection of the input boundary. For N -dimensional padding, use torch.nn.functional.pad() . Parameters : padding ( int , tuple ) – the size of the padding. If it is int , uses the same padding in all boundaries. If a 2- tuple , uses ( \(\text{padding\_left}\) , \(\text{padding\_right}\) ) Note that padding size should be less than the corresponding input dimension. Shape: Input: \((C, W_{in})\) or \((N, C, W_{in})\) . Output: \((C, W_{out})\) or \((N, C, W_{out})\) , where \(W_{out} = W_{in} + \text{padding\_left} + \text{padding\_right}\) Examples: >>> m = nn . ReflectionPad1d ( 2 ) >>> # xdoctest: +IGNORE_WANT("other tests seem to modify printing styles") >>> input = torch . arange ( 8 , dtype = torch . float ) . reshape ( 1 , 2 , 4 ) >>> input tensor([[[0., 1., 2., 3.], [4., 5., 6., 7.]]]) >>> m ( input ) tensor([[[2., 1., 0., 1., 2., 3., 2., 1.], [6., 5., 4., 5., 6., 7., 6., 5.]]]) >>> # using different paddings for different sides >>> m = nn . ReflectionPad1d (( 3 , 1 )) >>> m ( input ) tensor([[[3., 2., 1., 0., 1., 2., 3., 2.], [7., 6., 5., 4., 5., 6., 7., 6.]]]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # ReflectionPad2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.padding.ReflectionPad2d.html ReflectionPad2d class tensorplay.nn.modules.padding. ReflectionPad2d ( padding : int | tuple [ int , int , int , int ] ) [source] Pads the input tensor using the reflection of the input boundary. For N -dimensional padding, use torch.nn.functional.pad() . Parameters : padding ( int , tuple ) – the size of the padding. If it is int , uses the same padding in all boundaries. If a 4- tuple , uses ( \(\text{padding\_left}\) , \(\text{padding\_right}\) , \(\text{padding\_top}\) , \(\text{padding\_bottom}\) ) Note that padding size should be less than the corresponding input dimension. Shape: Input: \((N, C, H_{in}, W_{in})\) or \((C, H_{in}, W_{in})\) . Output: \((N, C, H_{out}, W_{out})\) or \((C, H_{out}, W_{out})\) where \(H_{out} = H_{in} + \text{padding\_top} + \text{padding\_bottom}\) \(W_{out} = W_{in} + \text{padding\_left} + \text{padding\_right}\) Examples: >>> # xdoctest: +IGNORE_WANT("not sure why xdoctest is choking on this") >>> m = nn . ReflectionPad2d ( 2 ) >>> input = torch . arange ( 9 , dtype = torch . float ) . reshape ( 1 , 1 , 3 , 3 ) >>> input tensor([[[[0., 1., 2.], [3., 4., 5.], [6., 7., 8.]]]]) >>> m ( input ) tensor([[[[8., 7., 6., 7., 8., 7., 6.], [5., 4., 3., 4., 5., 4., 3.], [2., 1., 0., 1., 2., 1., 0.], [5., 4., 3., 4., 5., 4., 3.], [8., 7., 6., 7., 8., 7., 6.], [5., 4., 3., 4., 5., 4., 3.], [2., 1., 0., 1., 2., 1., 0.]]]]) >>> # using different paddings for different sides >>> m = nn . ReflectionPad2d (( 1 , 1 , 2 , 0 )) >>> m ( input ) tensor([[[[7., 6., 7., 8., 7.], [4., 3., 4., 5., 4.], [1., 0., 1., 2., 1.], [4., 3., 4., 5., 4.], [7., 6., 7., 8., 7.]]]]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # ReflectionPad3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.padding.ReflectionPad3d.html ReflectionPad3d class tensorplay.nn.modules.padding. ReflectionPad3d ( padding : int | tuple [ int , int , int , int , int , int ] ) [source] Pads the input tensor using the reflection of the input boundary. For N -dimensional padding, use torch.nn.functional.pad() . Parameters : padding ( int , tuple ) – the size of the padding. If it is int , uses the same padding in all boundaries. If a 6- tuple , uses ( \(\text{padding\_left}\) , \(\text{padding\_right}\) , \(\text{padding\_top}\) , \(\text{padding\_bottom}\) , \(\text{padding\_front}\) , \(\text{padding\_back}\) ) Note that padding size should be less than the corresponding input dimension. Shape: Input: \((N, C, D_{in}, H_{in}, W_{in})\) or \((C, D_{in}, H_{in}, W_{in})\) . Output: \((N, C, D_{out}, H_{out}, W_{out})\) or \((C, D_{out}, H_{out}, W_{out})\) , where \(D_{out} = D_{in} + \text{padding\_front} + \text{padding\_back}\) \(H_{out} = H_{in} + \text{padding\_top} + \text{padding\_bottom}\) \(W_{out} = W_{in} + \text{padding\_left} + \text{padding\_right}\) Examples: >>> # xdoctest: +IGNORE_WANT("not sure why xdoctest is choking on this") >>> m = nn . ReflectionPad3d ( 1 ) >>> input = torch . arange ( 8 , dtype = torch . float ) . reshape ( 1 , 1 , 2 , 2 , 2 ) >>> m ( input ) tensor([[[[[7., 6., 7., 6.], [5., 4., 5., 4.], [7., 6., 7., 6.], [5., 4., 5., 4.]], [[3., 2., 3., 2.], [1., 0., 1., 0.], [3., 2., 3., 2.], [1., 0., 1., 0.]], [[7., 6., 7., 6.], [5., 4., 5., 4.], [7., 6., 7., 6.], [5., 4., 5., 4.]], [[3., 2., 3., 2.], [1., 0., 1., 0.], [3., 2., 3., 2.], [1., 0., 1., 0.]]]]]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # ReplicationPad1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.padding.ReplicationPad1d.html ReplicationPad1d class tensorplay.nn.modules.padding. ReplicationPad1d ( padding : int | tuple [ int , int ] ) [source] Pads the input tensor using replication of the input boundary. For N -dimensional padding, use torch.nn.functional.pad() . Parameters : padding ( int , tuple ) – the size of the padding. If it is int , uses the same padding in all boundaries. If a 2- tuple , uses ( \(\text{padding\_left}\) , \(\text{padding\_right}\) ) Note that the output dimensions must remain positive. Shape: Input: \((C, W_{in})\) or \((N, C, W_{in})\) . Output: \((C, W_{out})\) or \((N, C, W_{out})\) , where \(W_{out} = W_{in} + \text{padding\_left} + \text{padding\_right}\) Examples: >>> # xdoctest: +IGNORE_WANT("not sure why xdoctest is choking on this") >>> m = nn . ReplicationPad1d ( 2 ) >>> input = torch . arange ( 8 , dtype = torch . float ) . reshape ( 1 , 2 , 4 ) >>> input tensor([[[0., 1., 2., 3.], [4., 5., 6., 7.]]]) >>> m ( input ) tensor([[[0., 0., 0., 1., 2., 3., 3., 3.], [4., 4., 4., 5., 6., 7., 7., 7.]]]) >>> # using different paddings for different sides >>> m = nn . ReplicationPad1d (( 3 , 1 )) >>> m ( input ) tensor([[[0., 0., 0., 0., 1., 2., 3., 3.], [4., 4., 4., 4., 5., 6., 7., 7.]]]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # ReplicationPad2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.padding.ReplicationPad2d.html ReplicationPad2d class tensorplay.nn.modules.padding. ReplicationPad2d ( padding : int | tuple [ int , int , int , int ] ) [source] Pads the input tensor using replication of the input boundary. For N -dimensional padding, use torch.nn.functional.pad() . Parameters : padding ( int , tuple ) – the size of the padding. If it is int , uses the same padding in all boundaries. If a 4- tuple , uses ( \(\text{padding\_left}\) , \(\text{padding\_right}\) , \(\text{padding\_top}\) , \(\text{padding\_bottom}\) ) Note that the output dimensions must remain positive. Shape: Input: \((N, C, H_{in}, W_{in})\) or \((C, H_{in}, W_{in})\) . Output: \((N, C, H_{out}, W_{out})\) or \((C, H_{out}, W_{out})\) , where \(H_{out} = H_{in} + \text{padding\_top} + \text{padding\_bottom}\) \(W_{out} = W_{in} + \text{padding\_left} + \text{padding\_right}\) Examples: >>> m = nn . ReplicationPad2d ( 2 ) >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> input = torch . arange ( 9 , dtype = torch . float ) . reshape ( 1 , 1 , 3 , 3 ) >>> input tensor([[[[0., 1., 2.], [3., 4., 5.], [6., 7., 8.]]]]) >>> m ( input ) tensor([[[[0., 0., 0., 1., 2., 2., 2.], [0., 0., 0., 1., 2., 2., 2.], [0., 0., 0., 1., 2., 2., 2.], [3., 3., 3., 4., 5., 5., 5.], [6., 6., 6., 7., 8., 8., 8.], [6., 6., 6., 7., 8., 8., 8.], [6., 6., 6., 7., 8., 8., 8.]]]]) >>> # using different paddings for different sides >>> m = nn . ReplicationPad2d (( 1 , 1 , 2 , 0 )) >>> m ( input ) tensor([[[[0., 0., 1., 2., 2.], [0., 0., 1., 2., 2.], [0., 0., 1., 2., 2.], [3., 3., 4., 5., 5.], [6., 6., 7., 8., 8.]]]]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # ReplicationPad3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.padding.ReplicationPad3d.html ReplicationPad3d class tensorplay.nn.modules.padding. ReplicationPad3d ( padding : int | tuple [ int , int , int , int , int , int ] ) [source] Pads the input tensor using replication of the input boundary. For N -dimensional padding, use torch.nn.functional.pad() . Parameters : padding ( int , tuple ) – the size of the padding. If it is int , uses the same padding in all boundaries. If a 6- tuple , uses ( \(\text{padding\_left}\) , \(\text{padding\_right}\) , \(\text{padding\_top}\) , \(\text{padding\_bottom}\) , \(\text{padding\_front}\) , \(\text{padding\_back}\) ) Note that the output dimensions must remain positive. Shape: Input: \((N, C, D_{in}, H_{in}, W_{in})\) or \((C, D_{in}, H_{in}, W_{in})\) . Output: \((N, C, D_{out}, H_{out}, W_{out})\) or \((C, D_{out}, H_{out}, W_{out})\) , where \(D_{out} = D_{in} + \text{padding\_front} + \text{padding\_back}\) \(H_{out} = H_{in} + \text{padding\_top} + \text{padding\_bottom}\) \(W_{out} = W_{in} + \text{padding\_left} + \text{padding\_right}\) Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> m = nn . ReplicationPad3d ( 3 ) >>> input = torch . randn ( 16 , 3 , 8 , 320 , 480 ) >>> output = m ( input ) >>> # using different paddings for different sides >>> m = nn . ReplicationPad3d (( 3 , 3 , 6 , 6 , 1 , 1 )) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # ZeroPad1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.padding.ZeroPad1d.html ZeroPad1d class tensorplay.nn.modules.padding. ZeroPad1d ( padding : int | tuple [ int , int ] ) [source] Pads the input tensor boundaries with zero. For N -dimensional padding, use torch.nn.functional.pad() . Parameters : padding ( int , tuple ) – the size of the padding. If it is int , uses the same padding in both boundaries. If a 2- tuple , uses ( \(\text{padding\_left}\) , \(\text{padding\_right}\) ) Shape: Input: \((C, W_{in})\) or \((N, C, W_{in})\) . Output: \((C, W_{out})\) or \((N, C, W_{out})\) , where \(W_{out} = W_{in} + \text{padding\_left} + \text{padding\_right}\) Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> m = nn . ZeroPad1d ( 2 ) >>> input = torch . randn ( 1 , 2 , 4 ) >>> input tensor([[[-1.0491, -0.7152, -0.0749, 0.8530], [-1.3287, 1.8966, 0.1466, -0.2771]]]) >>> m ( input ) tensor([[[ 0.0000, 0.0000, -1.0491, -0.7152, -0.0749, 0.8530, 0.0000, 0.0000], [ 0.0000, 0.0000, -1.3287, 1.8966, 0.1466, -0.2771, 0.0000, 0.0000]]]) >>> m = nn . ZeroPad1d ( 2 ) >>> input = torch . randn ( 1 , 2 , 3 ) >>> input tensor([[[ 1.6616, 1.4523, -1.1255], [-3.6372, 0.1182, -1.8652]]]) >>> m ( input ) tensor([[[ 0.0000, 0.0000, 1.6616, 1.4523, -1.1255, 0.0000, 0.0000], [ 0.0000, 0.0000, -3.6372, 0.1182, -1.8652, 0.0000, 0.0000]]]) >>> # using different paddings for different sides >>> m = nn . ZeroPad1d (( 3 , 1 )) >>> m ( input ) tensor([[[ 0.0000, 0.0000, 0.0000, 1.6616, 1.4523, -1.1255, 0.0000], [ 0.0000, 0.0000, 0.0000, -3.6372, 0.1182, -1.8652, 0.0000]]]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str [source] Return the extra representation of the module. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # ZeroPad2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.padding.ZeroPad2d.html ZeroPad2d class tensorplay.nn.modules.padding. ZeroPad2d ( padding : int | tuple [ int , int , int , int ] ) [source] Pads the input tensor boundaries with zero. For N -dimensional padding, use torch.nn.functional.pad() . Parameters : padding ( int , tuple ) – the size of the padding. If it is int , uses the same padding in all boundaries. If a 4- tuple , uses ( \(\text{padding\_left}\) , \(\text{padding\_right}\) , \(\text{padding\_top}\) , \(\text{padding\_bottom}\) ) Shape: Input: \((N, C, H_{in}, W_{in})\) or \((C, H_{in}, W_{in})\) . Output: \((N, C, H_{out}, W_{out})\) or \((C, H_{out}, W_{out})\) , where \(H_{out} = H_{in} + \text{padding\_top} + \text{padding\_bottom}\) \(W_{out} = W_{in} + \text{padding\_left} + \text{padding\_right}\) Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> m = nn . ZeroPad2d ( 2 ) >>> input = torch . randn ( 1 , 1 , 3 , 3 ) >>> input tensor([[[[-0.1678, -0.4418, 1.9466], [ 0.9604, -0.4219, -0.5241], [-0.9162, -0.5436, -0.6446]]]]) >>> m ( input ) tensor([[[[ 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000], [ 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000], [ 0.0000, 0.0000, -0.1678, -0.4418, 1.9466, 0.0000, 0.0000], [ 0.0000, 0.0000, 0.9604, -0.4219, -0.5241, 0.0000, 0.0000], [ 0.0000, 0.0000, -0.9162, -0.5436, -0.6446, 0.0000, 0.0000], [ 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000], [ 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000]]]]) >>> # using different paddings for different sides >>> m = nn . ZeroPad2d (( 1 , 1 , 2 , 0 )) >>> m ( input ) tensor([[[[ 0.0000, 0.0000, 0.0000, 0.0000, 0.0000], [ 0.0000, 0.0000, 0.0000, 0.0000, 0.0000], [ 0.0000, -0.1678, -0.4418, 1.9466, 0.0000], [ 0.0000, 0.9604, -0.4219, -0.5241, 0.0000], [ 0.0000, -0.9162, -0.5436, -0.6446, 0.0000]]]]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str [source] Return the extra representation of the module. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # ZeroPad3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.padding.ZeroPad3d.html ZeroPad3d class tensorplay.nn.modules.padding. ZeroPad3d ( padding : int | tuple [ int , int , int , int , int , int ] ) [source] Pads the input tensor boundaries with zero. For N -dimensional padding, use torch.nn.functional.pad() . Parameters : padding ( int , tuple ) – the size of the padding. If it is int , uses the same padding in all boundaries. If a 6- tuple , uses ( \(\text{padding\_left}\) , \(\text{padding\_right}\) , \(\text{padding\_top}\) , \(\text{padding\_bottom}\) , \(\text{padding\_front}\) , \(\text{padding\_back}\) ) Shape: Input: \((N, C, D_{in}, H_{in}, W_{in})\) or \((C, D_{in}, H_{in}, W_{in})\) . Output: \((N, C, D_{out}, H_{out}, W_{out})\) or \((C, D_{out}, H_{out}, W_{out})\) , where \(D_{out} = D_{in} + \text{padding\_front} + \text{padding\_back}\) \(H_{out} = H_{in} + \text{padding\_top} + \text{padding\_bottom}\) \(W_{out} = W_{in} + \text{padding\_left} + \text{padding\_right}\) Examples: >>> m = nn . ZeroPad3d ( 3 ) >>> input = torch . randn ( 16 , 3 , 10 , 20 , 30 ) >>> output = m ( input ) >>> # using different paddings for different sides >>> m = nn . ZeroPad3d (( 3 , 3 , 6 , 6 , 0 , 1 )) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str [source] Return the extra representation of the module. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # PixelShuffle URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.pixelshuffle.PixelShuffle.html PixelShuffle class tensorplay.nn.modules.pixelshuffle. PixelShuffle ( upscale_factor : int ) [source] Rearrange elements in a tensor according to an upscaling factor. Rearranges elements in a tensor of shape \((*, C \times r^2, H, W)\) to a tensor of shape \((*, C, H \times r, W \times r)\) , where r is an upscale factor. This is useful for implementing efficient sub-pixel convolution with a stride of \(1/r\) . See the paper: Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network by Shi et al. (2016) for more details. Parameters : upscale_factor ( int ) – factor to increase spatial resolution by Shape: Input: \((*, C_{in}, H_{in}, W_{in})\) , where * is zero or more batch dimensions Output: \((*, C_{out}, H_{out}, W_{out})\) , where \[C_{out} = C_{in} \div \text{upscale\_factor}^2\] \[H_{out} = H_{in} \times \text{upscale\_factor}\] \[W_{out} = W_{in} \times \text{upscale\_factor}\] Examples: >>> pixel_shuffle = nn . PixelShuffle ( 3 ) >>> input = torch . randn ( 1 , 9 , 4 , 4 ) >>> output = pixel_shuffle ( input ) >>> print ( output . size ()) torch.Size([1, 1, 12, 12]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str [source] Return the extra representation of the module. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # PixelUnshuffle URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.pixelshuffle.PixelUnshuffle.html PixelUnshuffle class tensorplay.nn.modules.pixelshuffle. PixelUnshuffle ( downscale_factor : int ) [source] Reverse the PixelShuffle operation. Reverses the PixelShuffle operation by rearranging elements in a tensor of shape \((*, C, H \times r, W \times r)\) to a tensor of shape \((*, C \times r^2, H, W)\) , where r is a downscale factor. See the paper: Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network by Shi et al. (2016) for more details. Parameters : downscale_factor ( int ) – factor to decrease spatial resolution by Shape: Input: \((*, C_{in}, H_{in}, W_{in})\) , where * is zero or more batch dimensions Output: \((*, C_{out}, H_{out}, W_{out})\) , where \[C_{out} = C_{in} \times \text{downscale\_factor}^2\] \[H_{out} = H_{in} \div \text{downscale\_factor}\] \[W_{out} = W_{in} \div \text{downscale\_factor}\] Examples: >>> pixel_unshuffle = nn . PixelUnshuffle ( 3 ) >>> input = torch . randn ( 1 , 1 , 12 , 12 ) >>> output = pixel_unshuffle ( input ) >>> print ( output . size ()) torch.Size([1, 9, 4, 4]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str [source] Return the extra representation of the module. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # AdaptiveAvgPool1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.pooling.AdaptiveAvgPool1d.html AdaptiveAvgPool1d class tensorplay.nn.modules.pooling. AdaptiveAvgPool1d ( output_size : int | None | tuple [ int | None , ... ] ) [source] Applies a 1D adaptive average pooling over an input signal composed of several input planes. The output size is \(L_{out}\) , for any input size. The number of output features is equal to the number of input planes. Parameters : output_size – the target output size \(L_{out}\) . Shape: Input: \((N, C, L_{in})\) or \((C, L_{in})\) . Output: \((N, C, L_{out})\) or \((C, L_{out})\) , where \(L_{out}=\text{output\_size}\) . Examples >>> # target output size of 5 >>> m = nn . AdaptiveAvgPool1d ( 5 ) >>> input = tensorplay . randn ( 1 , 64 , 8 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # AdaptiveAvgPool2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.pooling.AdaptiveAvgPool2d.html AdaptiveAvgPool2d class tensorplay.nn.modules.pooling. AdaptiveAvgPool2d ( output_size : int | None | tuple [ int | None , ... ] ) [source] Applies a 2D adaptive average pooling over an input signal composed of several input planes. The output is of size H x W, for any input size. The number of output features is equal to the number of input planes. Parameters : output_size – the target output size of the image of the form H x W. Can be a tuple (H, W) or a single H for a square image H x H. H and W can be either a int , or None which means the size will be the same as that of the input. Shape: Input: \((N, C, H_{in}, W_{in})\) or \((C, H_{in}, W_{in})\) . Output: \((N, C, S_{0}, S_{1})\) or \((C, S_{0}, S_{1})\) , where \(S=\text{output\_size}\) . Examples >>> # target output size of 5x7 >>> m = nn . AdaptiveAvgPool2d (( 5 , 7 )) >>> input = tensorplay . randn ( 1 , 64 , 8 , 9 ) >>> output = m ( input ) >>> # target output size of 7x7 (square) >>> m = nn . AdaptiveAvgPool2d ( 7 ) >>> input = tensorplay . randn ( 1 , 64 , 10 , 9 ) >>> output = m ( input ) >>> # target output size of 10x7 >>> m = nn . AdaptiveAvgPool2d (( None , 7 )) >>> input = tensorplay . randn ( 1 , 64 , 10 , 9 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # AdaptiveAvgPool3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.pooling.AdaptiveAvgPool3d.html AdaptiveAvgPool3d class tensorplay.nn.modules.pooling. AdaptiveAvgPool3d ( output_size : int | None | tuple [ int | None , ... ] ) [source] Applies a 3D adaptive average pooling over an input signal composed of several input planes. The output is of size D x H x W, for any input size. The number of output features is equal to the number of input planes. Parameters : output_size – the target output size of the form D x H x W. Can be a tuple (D, H, W) or a single number D for a cube D x D x D. D, H and W can be either a int , or None which means the size will be the same as that of the input. Shape: Input: \((N, C, D_{in}, H_{in}, W_{in})\) or \((C, D_{in}, H_{in}, W_{in})\) . Output: \((N, C, S_{0}, S_{1}, S_{2})\) or \((C, S_{0}, S_{1}, S_{2})\) , where \(S=\text{output\_size}\) . Examples >>> # target output size of 5x7x9 >>> m = nn . AdaptiveAvgPool3d (( 5 , 7 , 9 )) >>> input = tensorplay . randn ( 1 , 64 , 8 , 9 , 10 ) >>> output = m ( input ) >>> # target output size of 7x7x7 (cube) >>> m = nn . AdaptiveAvgPool3d ( 7 ) >>> input = tensorplay . randn ( 1 , 64 , 10 , 9 , 8 ) >>> output = m ( input ) >>> # target output size of 7x9x8 >>> m = nn . AdaptiveAvgPool3d (( 7 , None , None )) >>> input = tensorplay . randn ( 1 , 64 , 10 , 9 , 8 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # AdaptiveMaxPool1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.pooling.AdaptiveMaxPool1d.html AdaptiveMaxPool1d class tensorplay.nn.modules.pooling. AdaptiveMaxPool1d ( output_size : int | None | tuple [ int | None , ... ] , return_indices : bool = False ) [source] Applies a 1D adaptive max pooling over an input signal composed of several input planes. The output size is \(L_{out}\) , for any input size. The number of output features is equal to the number of input planes. Parameters : output_size – the target output size \(L_{out}\) . return_indices – if True , will return the indices along with the outputs. Useful to pass to nn.MaxUnpool1d. Default: False Shape: Input: \((N, C, L_{in})\) or \((C, L_{in})\) . Output: \((N, C, L_{out})\) or \((C, L_{out})\) , where \(L_{out}=\text{output\_size}\) . Examples >>> # target output size of 5 >>> m = nn . AdaptiveMaxPool1d ( 5 ) >>> input = tensorplay . randn ( 1 , 64 , 8 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # AdaptiveMaxPool2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.pooling.AdaptiveMaxPool2d.html AdaptiveMaxPool2d class tensorplay.nn.modules.pooling. AdaptiveMaxPool2d ( output_size : int | None | tuple [ int | None , ... ] , return_indices : bool = False ) [source] Applies a 2D adaptive max pooling over an input signal composed of several input planes. The output is of size \(H_{out} \times W_{out}\) , for any input size. The number of output features is equal to the number of input planes. Parameters : output_size – the target output size of the image of the form \(H_{out} \times W_{out}\) . Can be a tuple \((H_{out}, W_{out})\) or a single \(H_{out}\) for a square image \(H_{out} \times H_{out}\) . \(H_{out}\) and \(W_{out}\) can be either a int , or None which means the size will be the same as that of the input. return_indices – if True , will return the indices along with the outputs. Useful to pass to nn.MaxUnpool2d. Default: False Shape: Input: \((N, C, H_{in}, W_{in})\) or \((C, H_{in}, W_{in})\) . Output: \((N, C, H_{out}, W_{out})\) or \((C, H_{out}, W_{out})\) , where \((H_{out}, W_{out})=\text{output\_size}\) . Examples >>> # target output size of 5x7 >>> m = nn . AdaptiveMaxPool2d (( 5 , 7 )) >>> input = tensorplay . randn ( 1 , 64 , 8 , 9 ) >>> output = m ( input ) >>> # target output size of 7x7 (square) >>> m = nn . AdaptiveMaxPool2d ( 7 ) >>> input = tensorplay . randn ( 1 , 64 , 10 , 9 ) >>> output = m ( input ) >>> # target output size of 10x7 >>> m = nn . AdaptiveMaxPool2d (( None , 7 )) >>> input = tensorplay . randn ( 1 , 64 , 10 , 9 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # AdaptiveMaxPool3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.pooling.AdaptiveMaxPool3d.html AdaptiveMaxPool3d class tensorplay.nn.modules.pooling. AdaptiveMaxPool3d ( output_size : int | None | tuple [ int | None , ... ] , return_indices : bool = False ) [source] Applies a 3D adaptive max pooling over an input signal composed of several input planes. The output is of size \(D_{out} \times H_{out} \times W_{out}\) , for any input size. The number of output features is equal to the number of input planes. Parameters : output_size – the target output size of the image of the form \(D_{out} \times H_{out} \times W_{out}\) . Can be a tuple \((D_{out}, H_{out}, W_{out})\) or a single \(D_{out}\) for a cube \(D_{out} \times D_{out} \times D_{out}\) . \(D_{out}\) , \(H_{out}\) and \(W_{out}\) can be either a int , or None which means the size will be the same as that of the input. return_indices – if True , will return the indices along with the outputs. Useful to pass to nn.MaxUnpool3d. Default: False Shape: Input: \((N, C, D_{in}, H_{in}, W_{in})\) or \((C, D_{in}, H_{in}, W_{in})\) . Output: \((N, C, D_{out}, H_{out}, W_{out})\) or \((C, D_{out}, H_{out}, W_{out})\) , where \((D_{out}, H_{out}, W_{out})=\text{output\_size}\) . Examples >>> # target output size of 5x7x9 >>> m = nn . AdaptiveMaxPool3d (( 5 , 7 , 9 )) >>> input = tensorplay . randn ( 1 , 64 , 8 , 9 , 10 ) >>> output = m ( input ) >>> # target output size of 7x7x7 (cube) >>> m = nn . AdaptiveMaxPool3d ( 7 ) >>> input = tensorplay . randn ( 1 , 64 , 10 , 9 , 8 ) >>> output = m ( input ) >>> # target output size of 7x9x8 >>> m = nn . AdaptiveMaxPool3d (( 7 , None , None )) >>> input = tensorplay . randn ( 1 , 64 , 10 , 9 , 8 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # AvgPool1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.pooling.AvgPool1d.html AvgPool1d class tensorplay.nn.modules.pooling. AvgPool1d ( kernel_size : int | tuple [ int ] , stride : int | tuple [ int ] | None = None , padding : int | tuple [ int ] = 0 , ceil_mode : bool = False , count_include_pad : bool = True ) [source] Applies a 1D average pooling over an input signal composed of several input planes. In the simplest case, the output value of the layer with input size \((N, C, L)\) , output \((N, C, L_{out})\) and kernel_size \(k\) can be precisely described as: \[\text{out}(N_i, C_j, l) = \frac{1}{k} \sum_{m=0}^{k-1} \text{input}(N_i, C_j, \text{stride} \times l + m)\] If padding is non-zero, then the input is implicitly zero-padded on both sides for padding number of points. Note When ceil_mode=True, sliding windows are allowed to go off-bounds if they start within the left padding or the input. Sliding windows that would start in the right padded region are ignored. Note pad should be at most half of effective kernel size. The parameters kernel_size , stride , padding can each be an int or a one-element tuple. Parameters : kernel_size – the size of the window stride – the stride of the window. Default value is kernel_size padding – implicit zero padding to be added on both sides ceil_mode – when True, will use ceil instead of floor to compute the output shape count_include_pad – when True, will include the zero-padding in the averaging calculation Shape: Input: \((N, C, L_{in})\) or \((C, L_{in})\) . Output: \((N, C, L_{out})\) or \((C, L_{out})\) , where \[L_{out} = \left\lfloor \frac{L_{in} + 2 \times \text{padding} - \text{kernel\_size}}{\text{stride}} + 1\right\rfloor\] Per the note above, if ceil_mode is True and \((L_{out} - 1) \times \text{stride} \geq L_{in} + \text{padding}\) , we skip the last window as it would start in the right padded region, resulting in \(L_{out}\) being reduced by one. Examples: >>> # pool with window of size=3, stride=2 >>> m = nn . AvgPool1d ( 3 , stride = 2 ) >>> m ( tensorplay . tensor ([[[ 1. , 2 , 3 , 4 , 5 , 6 , 7 ]]])) tensor([[[2., 4., 6.]]]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # AvgPool2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.pooling.AvgPool2d.html AvgPool2d class tensorplay.nn.modules.pooling. AvgPool2d ( kernel_size : int | tuple [ int , int ] , stride : int | tuple [ int , int ] | None = None , padding : int | tuple [ int , int ] = 0 , ceil_mode : bool = False , count_include_pad : bool = True , divisor_override : int | None = None ) [source] Applies a 2D average pooling over an input signal composed of several input planes. In the simplest case, the output value of the layer with input size \((N, C, H, W)\) , output \((N, C, H_{out}, W_{out})\) and kernel_size \((kH, kW)\) can be precisely described as: \[out(N_i, C_j, h, w) = \frac{1}{kH * kW} \sum_{m=0}^{kH-1} \sum_{n=0}^{kW-1} input(N_i, C_j, stride[0] \times h + m, stride[1] \times w + n)\] If padding is non-zero, then the input is implicitly zero-padded on both sides for padding number of points. Note When ceil_mode=True, sliding windows are allowed to go off-bounds if they start within the left padding or the input. Sliding windows that would start in the right padded region are ignored. Note pad should be at most half of effective kernel size. The parameters kernel_size , stride , padding can either be: a single int or a single-element tuple – in which case the same value is used for the height and width dimension a tuple of two ints – in which case, the first int is used for the height dimension, and the second int for the width dimension Parameters : kernel_size – the size of the window stride – the stride of the window. Default value is kernel_size padding – implicit zero padding to be added on both sides ceil_mode – when True, will use ceil instead of floor to compute the output shape count_include_pad – when True, will include the zero-padding in the averaging calculation divisor_override – if specified, it will be used as divisor, otherwise size of the pooling region will be used. Shape: Input: \((N, C, H_{in}, W_{in})\) or \((C, H_{in}, W_{in})\) . Output: \((N, C, H_{out}, W_{out})\) or \((C, H_{out}, W_{out})\) , where \[H_{out} = \left\lfloor\frac{H_{in} + 2 \times \text{padding}[0] - \text{kernel\_size}[0]}{\text{stride}[0]} + 1\right\rfloor\] \[W_{out} = \left\lfloor\frac{W_{in} + 2 \times \text{padding}[1] - \text{kernel\_size}[1]}{\text{stride}[1]} + 1\right\rfloor\] Per the note above, if ceil_mode is True and \((H_{out} - 1)\times \text{stride}[0]\geq H_{in} + \text{padding}[0]\) , we skip the last window as it would start in the bottom padded region, resulting in \(H_{out}\) being reduced by one. The same applies for \(W_{out}\) . Examples: >>> # pool of square window of size=3, stride=2 >>> m = nn . AvgPool2d ( 3 , stride = 2 ) >>> # pool of non-square window >>> m = nn . AvgPool2d (( 3 , 2 ), stride = ( 2 , 1 )) >>> input = tensorplay . randn ( 20 , 16 , 50 , 32 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # AvgPool3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.pooling.AvgPool3d.html AvgPool3d class tensorplay.nn.modules.pooling. AvgPool3d ( kernel_size : int | tuple [ int , int , int ] , stride : int | tuple [ int , int , int ] | None = None , padding : int | tuple [ int , int , int ] = 0 , ceil_mode : bool = False , count_include_pad : bool = True , divisor_override : int | None = None ) [source] Applies a 3D average pooling over an input signal composed of several input planes. In the simplest case, the output value of the layer with input size \((N, C, D, H, W)\) , output \((N, C, D_{out}, H_{out}, W_{out})\) and kernel_size \((kD, kH, kW)\) can be precisely described as: \[\begin{split}\begin{aligned} \text{out}(N_i, C_j, d, h, w) ={} & \sum_{k=0}^{kD-1} \sum_{m=0}^{kH-1} \sum_{n=0}^{kW-1} \\ & \frac{\text{input}(N_i, C_j, \text{stride}[0] \times d + k, \text{stride}[1] \times h + m, \text{stride}[2] \times w + n)} {kD \times kH \times kW} \end{aligned}\end{split}\] If padding is non-zero, then the input is implicitly zero-padded on all three sides for padding number of points. Note When ceil_mode=True, sliding windows are allowed to go off-bounds if they start within the left padding or the input. Sliding windows that would start in the right padded region are ignored. Note pad should be at most half of effective kernel size. The parameters kernel_size , stride can either be: a single int – in which case the same value is used for the depth, height and width dimension a tuple of three ints – in which case, the first int is used for the depth dimension, the second int for the height dimension and the third int for the width dimension Parameters : kernel_size – the size of the window stride – the stride of the window. Default value is kernel_size padding – implicit zero padding to be added on all three sides ceil_mode – when True, will use ceil instead of floor to compute the output shape count_include_pad – when True, will include the zero-padding in the averaging calculation divisor_override – if specified, it will be used as divisor, otherwise kernel_size will be used Shape: Input: \((N, C, D_{in}, H_{in}, W_{in})\) or \((C, D_{in}, H_{in}, W_{in})\) . Output: \((N, C, D_{out}, H_{out}, W_{out})\) or \((C, D_{out}, H_{out}, W_{out})\) , where \[D_{out} = \left\lfloor\frac{D_{in} + 2 \times \text{padding}[0] - \text{kernel\_size}[0]}{\text{stride}[0]} + 1\right\rfloor\] \[H_{out} = \left\lfloor\frac{H_{in} + 2 \times \text{padding}[1] - \text{kernel\_size}[1]}{\text{stride}[1]} + 1\right\rfloor\] \[W_{out} = \left\lfloor\frac{W_{in} + 2 \times \text{padding}[2] - \text{kernel\_size}[2]}{\text{stride}[2]} + 1\right\rfloor\] Per the note above, if ceil_mode is True and \((D_{out} - 1)\times \text{stride}[0]\geq D_{in} + \text{padding}[0]\) , we skip the last window as it would start in the padded region, resulting in \(D_{out}\) being reduced by one. The same applies for \(W_{out}\) and \(H_{out}\) . Examples: >>> # pool of square window of size=3, stride=2 >>> m = nn . AvgPool3d ( 3 , stride = 2 ) >>> # pool of non-square window >>> m = nn . AvgPool3d (( 3 , 2 , 2 ), stride = ( 2 , 1 , 2 )) >>> input = tensorplay . randn ( 20 , 16 , 50 , 44 , 31 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # FractionalMaxPool2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.pooling.FractionalMaxPool2d.html FractionalMaxPool2d class tensorplay.nn.modules.pooling. FractionalMaxPool2d ( kernel_size : int | tuple [ int , int ] , output_size : int | tuple [ int , int ] | None = None , output_ratio : float | tuple [ float , float ] | None = None , return_indices : bool = False , _random_samples = None ) [source] Applies a 2D fractional max pooling over an input signal composed of several input planes. Fractional MaxPooling is described in detail in the paper Fractional MaxPooling by Ben Graham The max-pooling operation is applied in \(kH \times kW\) regions by a stochastic step size determined by the target output size. The number of output features is equal to the number of input planes. Note Exactly one of output_size or output_ratio must be defined. Parameters : kernel_size – the size of the window to take a max over. Can be a single number k (for a square kernel of k x k) or a tuple (kh, kw) output_size – the target output size of the image of the form oH x oW . Can be a tuple (oH, oW) or a single number oH for a square image oH x oH . Note that we must have \(kH + oH - 1 <= H_{in}\) and \(kW + oW - 1 <= W_{in}\) output_ratio – If one wants to have an output size as a ratio of the input size, this option can be given. This has to be a number or tuple in the range (0, 1). Note that we must have \(kH + (output\_ratio\_H * H_{in}) - 1 <= H_{in}\) and \(kW + (output\_ratio\_W * W_{in}) - 1 <= W_{in}\) return_indices – if True , will return the indices along with the outputs. Useful to pass to nn.MaxUnpool2d() . Default: False Shape: Input: \((N, C, H_{in}, W_{in})\) or \((C, H_{in}, W_{in})\) . Output: \((N, C, H_{out}, W_{out})\) or \((C, H_{out}, W_{out})\) , where \((H_{out}, W_{out})=\text{output\_size}\) or \((H_{out}, W_{out})=\text{output\_ratio} \times (H_{in}, W_{in})\) . Examples >>> # pool of square window of size=3, and target output size 13x12 >>> m = nn . FractionalMaxPool2d ( 3 , output_size = ( 13 , 12 )) >>> # pool of square window and target output size being half of input image size >>> m = nn . FractionalMaxPool2d ( 3 , output_ratio = ( 0.5 , 0.5 )) >>> input = tensorplay . randn ( 20 , 16 , 50 , 32 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # FractionalMaxPool3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.pooling.FractionalMaxPool3d.html FractionalMaxPool3d class tensorplay.nn.modules.pooling. FractionalMaxPool3d ( kernel_size : int | tuple [ int , int , int ] , output_size : int | tuple [ int , int , int ] | None = None , output_ratio : float | tuple [ float , float , float ] | None = None , return_indices : bool = False , _random_samples = None ) [source] Applies a 3D fractional max pooling over an input signal composed of several input planes. Fractional MaxPooling is described in detail in the paper Fractional MaxPooling by Ben Graham The max-pooling operation is applied in \(kT \times kH \times kW\) regions by a stochastic step size determined by the target output size. The number of output features is equal to the number of input planes. Note Exactly one of output_size or output_ratio must be defined. Parameters : kernel_size – the size of the window to take a max over. Can be a single number k (for a square kernel of k x k x k ) or a tuple (kt x kh x kw) , k must greater than 0. output_size – the target output size of the image of the form oT x oH x oW . Can be a tuple (oT, oH, oW) or a single number oH for a square image oH x oH x oH output_ratio – If one wants to have an output size as a ratio of the input size, this option can be given. This has to be a number or tuple in the range (0, 1) return_indices – if True , will return the indices along with the outputs. Useful to pass to nn.MaxUnpool3d() . Default: False Shape: Input: \((N, C, T_{in}, H_{in}, W_{in})\) or \((C, T_{in}, H_{in}, W_{in})\) . Output: \((N, C, T_{out}, H_{out}, W_{out})\) or \((C, T_{out}, H_{out}, W_{out})\) , where \((T_{out}, H_{out}, W_{out})=\text{output\_size}\) or \((T_{out}, H_{out}, W_{out})=\text{output\_ratio} \times (T_{in}, H_{in}, W_{in})\) Examples >>> # pool of cubic window of size=3, and target output size 13x12x11 >>> m = nn . FractionalMaxPool3d ( 3 , output_size = ( 13 , 12 , 11 )) >>> # pool of cubic window and target output size being half of input size >>> m = nn . FractionalMaxPool3d ( 3 , output_ratio = ( 0.5 , 0.5 , 0.5 )) >>> input = tensorplay . randn ( 20 , 16 , 50 , 32 , 16 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # LPPool1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.pooling.LPPool1d.html LPPool1d class tensorplay.nn.modules.pooling. LPPool1d ( norm_type : float , kernel_size : int | tuple [ int , ... ] , stride : int | tuple [ int , ... ] | None = None , ceil_mode : bool = False ) [source] Applies a 1D power-average pooling over an input signal composed of several input planes. On each window, the function computed is: \[f(X) = \sqrt[p]{\sum_{x \in X} x^{p}}\] At p = \(\infty\) , one gets Max Pooling At p = 1, one gets Sum Pooling (which is proportional to Average Pooling) Note If the sum to the power of p is zero, the gradient of this function is not defined. This implementation will set the gradient to zero in this case. Parameters : kernel_size – a single int, the size of the window stride – a single int, the stride of the window. Default value is kernel_size ceil_mode – when True, will use ceil instead of floor to compute the output shape Shape: Input: \((N, C, L_{in})\) or \((C, L_{in})\) . Output: \((N, C, L_{out})\) or \((C, L_{out})\) , where \[L_{out} = \left\lfloor\frac{L_{in} - \text{kernel\_size}}{\text{stride}} + 1\right\rfloor\] Examples:: >>> # power-2 pool of window of length 3, with stride 2. >>> m = nn . LPPool1d ( 2 , 3 , stride = 2 ) >>> input = tensorplay . randn ( 20 , 16 , 50 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # LPPool2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.pooling.LPPool2d.html LPPool2d class tensorplay.nn.modules.pooling. LPPool2d ( norm_type : float , kernel_size : int | tuple [ int , ... ] , stride : int | tuple [ int , ... ] | None = None , ceil_mode : bool = False ) [source] Applies a 2D power-average pooling over an input signal composed of several input planes. On each window, the function computed is: \[f(X) = \sqrt[p]{\sum_{x \in X} x^{p}}\] At p = \(\infty\) , one gets Max Pooling At p = 1, one gets Sum Pooling (which is proportional to average pooling) The parameters kernel_size , stride can either be: a single int – in which case the same value is used for the height and width dimension a tuple of two ints – in which case, the first int is used for the height dimension, and the second int for the width dimension Note If the sum to the power of p is zero, the gradient of this function is not defined. This implementation will set the gradient to zero in this case. Parameters : kernel_size – the size of the window stride – the stride of the window. Default value is kernel_size ceil_mode – when True, will use ceil instead of floor to compute the output shape Shape: Input: \((N, C, H_{in}, W_{in})\) or \((C, H_{in}, W_{in})\) . Output: \((N, C, H_{out}, W_{out})\) or \((C, H_{out}, W_{out})\) , where \[H_{out} = \left\lfloor\frac{H_{in} - \text{kernel\_size}[0]}{\text{stride}[0]} + 1\right\rfloor\] \[W_{out} = \left\lfloor\frac{W_{in} - \text{kernel\_size}[1]}{\text{stride}[1]} + 1\right\rfloor\] Examples: >>> # power-2 pool of square window of size=3, stride=2 >>> m = nn . LPPool2d ( 2 , 3 , stride = 2 ) >>> # pool of non-square window of power 1.2 >>> m = nn . LPPool2d ( 1.2 , ( 3 , 2 ), stride = ( 2 , 1 )) >>> input = tensorplay . randn ( 20 , 16 , 50 , 32 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # LPPool3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.pooling.LPPool3d.html LPPool3d class tensorplay.nn.modules.pooling. LPPool3d ( norm_type : float , kernel_size : int | tuple [ int , ... ] , stride : int | tuple [ int , ... ] | None = None , ceil_mode : bool = False ) [source] Applies a 3D power-average pooling over an input signal composed of several input planes. On each window, the function computed is: \[f(X) = \sqrt[p]{\sum_{x \in X} x^{p}}\] At p = \(\infty\) , one gets Max Pooling At p = 1, one gets Sum Pooling (which is proportional to average pooling) The parameters kernel_size , stride can either be: a single int – in which case the same value is used for the height, width and depth dimension a tuple of three ints – in which case, the first int is used for the depth dimension, the second int for the height dimension and the third int for the width dimension Note If the sum to the power of p is zero, the gradient of this function is not defined. This implementation will set the gradient to zero in this case. Parameters : kernel_size – the size of the window stride – the stride of the window. Default value is kernel_size ceil_mode – when True, will use ceil instead of floor to compute the output shape Shape: Input: \((N, C, D_{in}, H_{in}, W_{in})\) or \((C, D_{in}, H_{in}, W_{in})\) . Output: \((N, C, D_{out}, H_{out}, W_{out})\) or \((C, D_{out}, H_{out}, W_{out})\) , where \[D_{out} = \left\lfloor\frac{D_{in} - \text{kernel\_size}[0]}{\text{stride}[0]} + 1\right\rfloor\] \[H_{out} = \left\lfloor\frac{H_{in} - \text{kernel\_size}[1]}{\text{stride}[1]} + 1\right\rfloor\] \[W_{out} = \left\lfloor\frac{W_{in} - \text{kernel\_size}[2]}{\text{stride}[2]} + 1\right\rfloor\] Examples: >>> # power-2 pool of square window of size=3, stride=2 >>> m = nn . LPPool3d ( 2 , 3 , stride = 2 ) >>> # pool of non-square window of power 1.2 >>> m = nn . LPPool3d ( 1.2 , ( 3 , 2 , 2 ), stride = ( 2 , 1 , 2 )) >>> input = tensorplay . randn ( 20 , 16 , 50 , 44 , 31 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # MaxPool1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.pooling.MaxPool1d.html MaxPool1d class tensorplay.nn.modules.pooling. MaxPool1d ( kernel_size : int | tuple [ int , ... ] , stride : int | tuple [ int , ... ] | None = None , padding : int | tuple [ int , ... ] = 0 , dilation : int | tuple [ int , ... ] = 1 , return_indices : bool = False , ceil_mode : bool = False ) [source] Applies a 1D max pooling over an input signal composed of several input planes. In the simplest case, the output value of the layer with input size \((N, C, L)\) and output \((N, C, L_{out})\) can be precisely described as: \[out(N_i, C_j, k) = \max_{m=0, \ldots, \text{kernel\_size} - 1} input(N_i, C_j, stride \times k + m)\] If padding is non-zero, then the input is implicitly padded with negative infinity on both sides for padding number of points. dilation is the stride between the elements within the sliding window. This `link`_ has a nice visualization of the pooling parameters. Note When ceil_mode=True, sliding windows are allowed to go off-bounds if they start within the left padding or the input. Sliding windows that would start in the right padded region are ignored. Parameters : kernel_size – The size of the sliding window, must be > 0. stride – The stride of the sliding window, must be > 0. Default value is kernel_size . padding – Implicit negative infinity padding to be added on both sides, must be >= 0 and <= kernel_size / 2. dilation – The stride between elements within a sliding window, must be > 0. return_indices – If True , will return the argmax along with the max values. Useful for tensorplay.nn.MaxUnpool1d later ceil_mode – If True , will use ceil instead of floor to compute the output shape. This ensures that every element in the input tensor is covered by a sliding window. Shape: Input: \((N, C, L_{in})\) or \((C, L_{in})\) . Output: \((N, C, L_{out})\) or \((C, L_{out})\) , where ceil_mode = False \[L_{out} = \left\lfloor \frac{L_{in} + 2 \times \text{padding} - \text{dilation} \times (\text{kernel\_size} - 1) - 1}{\text{stride}}\right\rfloor + 1\] where ceil_mode = True \[L_{out} = \left\lceil \frac{L_{in} + 2 \times \text{padding} - \text{dilation} \times (\text{kernel\_size} - 1) - 1 + (stride - 1)}{\text{stride}}\right\rceil + 1\] Ensure that the last pooling starts inside the image, make \(L_{out} = L_{out} - 1\) when \((L_{out} - 1) * \text{stride} >= L_{in} + \text{padding}\) . Examples: >>> # pool of size=3, stride=2 >>> m = nn . MaxPool1d ( 3 , stride = 2 ) >>> input = tensorplay . randn ( 20 , 16 , 50 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # MaxPool2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.pooling.MaxPool2d.html MaxPool2d class tensorplay.nn.modules.pooling. MaxPool2d ( kernel_size : int | tuple [ int , ... ] , stride : int | tuple [ int , ... ] | None = None , padding : int | tuple [ int , ... ] = 0 , dilation : int | tuple [ int , ... ] = 1 , return_indices : bool = False , ceil_mode : bool = False ) [source] Applies a 2D max pooling over an input signal composed of several input planes. In the simplest case, the output value of the layer with input size \((N, C, H, W)\) , output \((N, C, H_{out}, W_{out})\) and kernel_size \((kH, kW)\) can be precisely described as: \[\begin{split}\begin{aligned} out(N_i, C_j, h, w) ={} & \max_{m=0, \ldots, kH-1} \max_{n=0, \ldots, kW-1} \\ & \text{input}(N_i, C_j, \text{stride[0]} \times h + m, \text{stride[1]} \times w + n) \end{aligned}\end{split}\] If padding is non-zero, then the input is implicitly padded with negative infinity on both sides for padding number of points. dilation controls the spacing between the kernel points. It is harder to describe, but this `link`_ has a nice visualization of what dilation does. Note When ceil_mode=True, sliding windows are allowed to go off-bounds if they start within the left padding or the input. Sliding windows that would start in the right padded region are ignored. The parameters kernel_size , stride , padding , dilation can either be: a single int – in which case the same value is used for the height and width dimension a tuple of two ints – in which case, the first int is used for the height dimension, and the second int for the width dimension Parameters : kernel_size – the size of the window to take a max over stride – the stride of the window. Default value is kernel_size padding – Implicit negative infinity padding to be added on both sides dilation – a parameter that controls the stride of elements in the window return_indices – if True , will return the max indices along with the outputs. Useful for tensorplay.nn.MaxUnpool2d later ceil_mode – when True, will use ceil instead of floor to compute the output shape Shape: Input: \((N, C, H_{in}, W_{in})\) or \((C, H_{in}, W_{in})\) Output: \((N, C, H_{out}, W_{out})\) or \((C, H_{out}, W_{out})\) , where \[H_{out} = \left\lfloor\frac{H_{in} + 2 * \text{padding[0]} - \text{dilation[0]} \times (\text{kernel\_size[0]} - 1) - 1}{\text{stride[0]}} + 1\right\rfloor\] \[W_{out} = \left\lfloor\frac{W_{in} + 2 * \text{padding[1]} - \text{dilation[1]} \times (\text{kernel\_size[1]} - 1) - 1}{\text{stride[1]}} + 1\right\rfloor\] Examples: >>> # pool of square window of size=3, stride=2 >>> m = nn . MaxPool2d ( 3 , stride = 2 ) >>> # pool of non-square window >>> m = nn . MaxPool2d (( 3 , 2 ), stride = ( 2 , 1 )) >>> input = tensorplay . randn ( 20 , 16 , 50 , 32 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # MaxPool3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.pooling.MaxPool3d.html MaxPool3d class tensorplay.nn.modules.pooling. MaxPool3d ( kernel_size : int | tuple [ int , ... ] , stride : int | tuple [ int , ... ] | None = None , padding : int | tuple [ int , ... ] = 0 , dilation : int | tuple [ int , ... ] = 1 , return_indices : bool = False , ceil_mode : bool = False ) [source] Applies a 3D max pooling over an input signal composed of several input planes. In the simplest case, the output value of the layer with input size \((N, C, D, H, W)\) , output \((N, C, D_{out}, H_{out}, W_{out})\) and kernel_size \((kD, kH, kW)\) can be precisely described as: \[\begin{split}\begin{aligned} \text{out}(N_i, C_j, d, h, w) ={} & \max_{k=0, \ldots, kD-1} \max_{m=0, \ldots, kH-1} \max_{n=0, \ldots, kW-1} \\ & \text{input}(N_i, C_j, \text{stride[0]} \times d + k, \text{stride[1]} \times h + m, \text{stride[2]} \times w + n) \end{aligned}\end{split}\] If padding is non-zero, then the input is implicitly padded with negative infinity on both sides for padding number of points. dilation controls the spacing between the kernel points. It is harder to describe, but this `link`_ has a nice visualization of what dilation does. Note When ceil_mode=True, sliding windows are allowed to go off-bounds if they start within the left padding or the input. Sliding windows that would start in the right padded region are ignored. The parameters kernel_size , stride , padding , dilation can either be: a single int – in which case the same value is used for the depth, height and width dimension a tuple of three ints – in which case, the first int is used for the depth dimension, the second int for the height dimension and the third int for the width dimension Parameters : kernel_size – the size of the window to take a max over stride – the stride of the window. Default value is kernel_size padding – Implicit negative infinity padding to be added on all three sides dilation – a parameter that controls the stride of elements in the window return_indices – if True , will return the max indices along with the outputs. Useful for tensorplay.nn.MaxUnpool3d later ceil_mode – when True, will use ceil instead of floor to compute the output shape Shape: Input: \((N, C, D_{in}, H_{in}, W_{in})\) or \((C, D_{in}, H_{in}, W_{in})\) . Output: \((N, C, D_{out}, H_{out}, W_{out})\) or \((C, D_{out}, H_{out}, W_{out})\) , where \[D_{out} = \left\lfloor\frac{D_{in} + 2 \times \text{padding}[0] - \text{dilation}[0] \times (\text{kernel\_size}[0] - 1) - 1}{\text{stride}[0]} + 1\right\rfloor\] \[H_{out} = \left\lfloor\frac{H_{in} + 2 \times \text{padding}[1] - \text{dilation}[1] \times (\text{kernel\_size}[1] - 1) - 1}{\text{stride}[1]} + 1\right\rfloor\] \[W_{out} = \left\lfloor\frac{W_{in} + 2 \times \text{padding}[2] - \text{dilation}[2] \times (\text{kernel\_size}[2] - 1) - 1}{\text{stride}[2]} + 1\right\rfloor\] Examples: >>> # pool of square window of size=3, stride=2 >>> m = nn . MaxPool3d ( 3 , stride = 2 ) >>> # pool of non-square window >>> m = nn . MaxPool3d (( 3 , 2 , 2 ), stride = ( 2 , 1 , 2 )) >>> input = tensorplay . randn ( 20 , 16 , 50 , 44 , 31 ) >>> output = m ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # MaxUnpool1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.pooling.MaxUnpool1d.html MaxUnpool1d class tensorplay.nn.modules.pooling. MaxUnpool1d ( kernel_size : int | tuple [ int ] , stride : int | tuple [ int ] | None = None , padding : int | tuple [ int ] = 0 ) [source] Computes a partial inverse of MaxPool1d . MaxPool1d is not fully invertible, since the non-maximal values are lost. MaxUnpool1d takes in as input the output of MaxPool1d including the indices of the maximal values and computes a partial inverse in which all non-maximal values are set to zero. Note This operation may behave nondeterministically when the input indices has repeat values. Note MaxPool1d can map several input sizes to the same output sizes. Hence, the inversion process can get ambiguous. To accommodate this, you can provide the needed output size as an additional argument output_size in the forward call. See the Inputs and Example below. Parameters : kernel_size ( int or tuple ) – Size of the max pooling window. stride ( int or tuple ) – Stride of the max pooling window. It is set to kernel_size by default. padding ( int or tuple ) – Padding that was added to the input Inputs: input : the input Tensor to invert indices : the indices given out by MaxPool1d output_size (optional): the targeted output size Shape: Input: \((N, C, H_{in})\) or \((C, H_{in})\) . Output: \((N, C, H_{out})\) or \((C, H_{out})\) , where \[H_{out} = (H_{in} - 1) \times \text{stride}[0] - 2 \times \text{padding}[0] + \text{kernel\_size}[0]\] or as given by output_size in the call operator Example: >>> # xdoctest: +IGNORE_WANT("do other tests modify the global state?") >>> pool = nn . MaxPool1d ( 2 , stride = 2 , return_indices = True ) >>> unpool = nn . MaxUnpool1d ( 2 , stride = 2 ) >>> input = tensorplay . tensor ([[[ 1. , 2 , 3 , 4 , 5 , 6 , 7 , 8 ]]]) >>> output , indices = pool ( input ) >>> unpool ( output , indices ) tensor([[[ 0., 2., 0., 4., 0., 6., 0., 8.]]]) >>> # Example showcasing the use of output_size >>> input = tensorplay . tensor ([[[ 1. , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 ]]]) >>> output , indices = pool ( input ) >>> unpool ( output , indices , output_size = input . size ()) tensor([[[ 0., 2., 0., 4., 0., 6., 0., 8., 0.]]]) >>> unpool ( output , indices ) tensor([[[ 0., 2., 0., 4., 0., 6., 0., 8.]]]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase , indices : TensorBase , output_size : list [ int ] | None = None ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # MaxUnpool2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.pooling.MaxUnpool2d.html MaxUnpool2d class tensorplay.nn.modules.pooling. MaxUnpool2d ( kernel_size : int | tuple [ int , int ] , stride : int | tuple [ int , int ] | None = None , padding : int | tuple [ int , int ] = 0 ) [source] Computes a partial inverse of MaxPool2d . MaxPool2d is not fully invertible, since the non-maximal values are lost. MaxUnpool2d takes in as input the output of MaxPool2d including the indices of the maximal values and computes a partial inverse in which all non-maximal values are set to zero. Note This operation may behave nondeterministically when the input indices has repeat values. Note MaxPool2d can map several input sizes to the same output sizes. Hence, the inversion process can get ambiguous. To accommodate this, you can provide the needed output size as an additional argument output_size in the forward call. See the Inputs and Example below. Parameters : kernel_size ( int or tuple ) – Size of the max pooling window. stride ( int or tuple ) – Stride of the max pooling window. It is set to kernel_size by default. padding ( int or tuple ) – Padding that was added to the input Inputs: input : the input Tensor to invert indices : the indices given out by MaxPool2d output_size (optional): the targeted output size Shape: Input: \((N, C, H_{in}, W_{in})\) or \((C, H_{in}, W_{in})\) . Output: \((N, C, H_{out}, W_{out})\) or \((C, H_{out}, W_{out})\) , where \[H_{out} = (H_{in} - 1) \times \text{stride[0]} - 2 \times \text{padding[0]} + \text{kernel\_size[0]}\] \[W_{out} = (W_{in} - 1) \times \text{stride[1]} - 2 \times \text{padding[1]} + \text{kernel\_size[1]}\] or as given by output_size in the call operator Example: >>> pool = nn . MaxPool2d ( 2 , stride = 2 , return_indices = True ) >>> unpool = nn . MaxUnpool2d ( 2 , stride = 2 ) >>> input = tensorplay . tensor ([[[[ 1. , 2. , 3. , 4. ], [ 5., 6., 7., 8.], [ 9., 10., 11., 12.], [13., 14., 15., 16.]]]]) >>> output , indices = pool ( input ) >>> unpool ( output , indices ) tensor([[[[ 0., 0., 0., 0.], [ 0., 6., 0., 8.], [ 0., 0., 0., 0.], [ 0., 14., 0., 16.]]]]) >>> # Now using output_size to resolve an ambiguous size for the inverse >>> input = tensorplay . tensor ([[[[ 1. , 2. , 3. , 4. , 5. ], [ 6., 7., 8., 9., 10.], [11., 12., 13., 14., 15.], [16., 17., 18., 19., 20.]]]]) >>> output , indices = pool ( input ) >>> # This call will not work without specifying output_size >>> unpool ( output , indices , output_size = input . size ()) tensor([[[[ 0., 0., 0., 0., 0.], [ 0., 7., 0., 9., 0.], [ 0., 0., 0., 0., 0.], [ 0., 17., 0., 19., 0.]]]]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase , indices : TensorBase , output_size : list [ int ] | None = None ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # MaxUnpool3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.pooling.MaxUnpool3d.html MaxUnpool3d class tensorplay.nn.modules.pooling. MaxUnpool3d ( kernel_size : int | tuple [ int , int , int ] , stride : int | tuple [ int , int , int ] | None = None , padding : int | tuple [ int , int , int ] = 0 ) [source] Computes a partial inverse of MaxPool3d . MaxPool3d is not fully invertible, since the non-maximal values are lost. MaxUnpool3d takes in as input the output of MaxPool3d including the indices of the maximal values and computes a partial inverse in which all non-maximal values are set to zero. Note This operation may behave nondeterministically when the input indices has repeat values. Note MaxPool3d can map several input sizes to the same output sizes. Hence, the inversion process can get ambiguous. To accommodate this, you can provide the needed output size as an additional argument output_size in the forward call. See the Inputs section below. Parameters : kernel_size ( int or tuple ) – Size of the max pooling window. stride ( int or tuple ) – Stride of the max pooling window. It is set to kernel_size by default. padding ( int or tuple ) – Padding that was added to the input Inputs: input : the input Tensor to invert indices : the indices given out by MaxPool3d output_size (optional): the targeted output size Shape: Input: \((N, C, D_{in}, H_{in}, W_{in})\) or \((C, D_{in}, H_{in}, W_{in})\) . Output: \((N, C, D_{out}, H_{out}, W_{out})\) or \((C, D_{out}, H_{out}, W_{out})\) , where \[D_{out} = (D_{in} - 1) \times \text{stride[0]} - 2 \times \text{padding[0]} + \text{kernel\_size[0]}\] \[H_{out} = (H_{in} - 1) \times \text{stride[1]} - 2 \times \text{padding[1]} + \text{kernel\_size[1]}\] \[W_{out} = (W_{in} - 1) \times \text{stride[2]} - 2 \times \text{padding[2]} + \text{kernel\_size[2]}\] or as given by output_size in the call operator Example: >>> # pool of square window of size=3, stride=2 >>> pool = nn . MaxPool3d ( 3 , stride = 2 , return_indices = True ) >>> unpool = nn . MaxUnpool3d ( 3 , stride = 2 ) >>> output , indices = pool ( tensorplay . randn ( 20 , 16 , 51 , 33 , 15 )) >>> unpooled_output = unpool ( output , indices ) >>> unpooled_output . size () tensorplay.Size([20, 16, 51, 33, 15]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase , indices : TensorBase , output_size : list [ int ] | None = None ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # GRU URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.rnn.GRU.html GRU class tensorplay.nn.modules.rnn. GRU ( * args , ** kwargs ) [source] __init__(input_size, hidden_size, num_layers=1, bias=True, batch_first=False, dropout=0.0, bidirectional=False, device=None, dtype=None) Apply a multi-layer gated recurrent unit (GRU) RNN to an input sequence. For each element in the input sequence, each layer computes the following function: \[\begin{split}\begin{array}{ll} \\ r_t = \sigma(W_{ir} x_t + b_{ir} + W_{hr} h_{(t-1)} + b_{hr}) \\ z_t = \sigma(W_{iz} x_t + b_{iz} + W_{hz} h_{(t-1)} + b_{hz}) \\ n_t = \tanh(W_{in} x_t + b_{in} + r_t \odot (W_{hn} h_{(t-1)} + b_{hn})) \\ h_t = (1 - z_t) \odot n_t + z_t \odot h_{(t-1)} \\ \end{array}\end{split}\] where \(h_t\) is the hidden state at time t , \(x_t\) is the input at time t , \(h_{(t-1)}\) is the hidden state of the layer at time t-1 or the initial hidden state at time 0, and \(r_t\) , \(z_t\) , \(n_t\) are the reset, update, and new gates, respectively. \(\sigma\) is the sigmoid function, and \(\odot\) is the Hadamard product. Parameters : input_size – The number of expected features in the input x hidden_size – The number of features in the hidden state h num_layers – Number of recurrent layers. E.g., setting num_layers=2 would mean stacking two GRUs together to form a stacked GRU , with the second GRU taking in outputs of the first GRU and computing the final results. Default: 1 bias – If False , then the layer does not use bias weights b_ih and b_hh . Default: True batch_first – If True , then the input and output tensors are provided as (batch, seq, feature) instead of (seq, batch, feature) . Note that this does not apply to hidden or cell states. See the Inputs/Outputs sections below for details. Default: False dropout – If non-zero, introduces a Dropout layer on the outputs of each GRU layer except the last layer, with dropout probability equal to dropout . Default: 0 bidirectional – If True , becomes a bidirectional GRU. Default: False Inputs: input, h_0 input : tensor of shape \((L, H_{in})\) for unbatched input, \((L, N, H_{in})\) when batch_first=False or \((N, L, H_{in})\) when batch_first=True containing the features of the input sequence. The input can also be a packed variable length sequence. See tensorplay.nn.utils.rnn.pack_padded_sequence() for details. h_0 : tensor of shape \((D * \text{num\_layers}, H_{out})\) for unbatched input or \((D * \text{num\_layers}, N, H_{out})\) containing the initial hidden state for each element in the input sequence. Defaults to zeros if not provided. where: \[\begin{split}\begin{aligned} N ={} & \text{batch size} \\ L ={} & \text{sequence length} \\ D ={} & 2 \text{ if bidirectional=True otherwise } 1 \\ H_{in} ={} & \text{input\_size} \\ H_{out} ={} & \text{hidden\_size} \end{aligned}\end{split}\] Outputs: output, h_n output : tensor of shape \((L, D * H_{out})\) for unbatched input, \((L, N, D * H_{out})\) when batch_first=False or \((N, L, D * H_{out})\) when batch_first=True containing the output features (h_t) from the last layer of the GRU, for each t . If a PackedSequence has been given as the input, the output will also be a packed sequence. h_n : tensor of shape \((D * \text{num\_layers}, H_{out})\) for unbatched input or \((D * \text{num\_layers}, N, H_{out})\) containing the final hidden state for each element in the batch. weight_ih_l [ k ] the learnable input-hidden weights of the k-th layer \((W_{ir}|W_{iz}|W_{in})\) , of shape (3*hidden_size, input_size) for k = 0 . Otherwise, the shape is (3*hidden_size, num_directions * hidden_size) weight_hh_l [ k ] the learnable hidden-hidden weights of the k-th layer \((W_{hr}|W_{hz}|W_{hn})\) , of shape (3*hidden_size, hidden_size) bias_ih_l [ k ] the learnable input-hidden bias of the k-th layer \((b_{ir}|b_{iz}|b_{in})\) , of shape (3*hidden_size) bias_hh_l [ k ] the learnable hidden-hidden bias of the k-th layer \((b_{hr}|b_{hz}|b_{hn})\) , of shape (3*hidden_size) Note All the weights and biases are initialized from \(\mathcal{U}(-\sqrt{k}, \sqrt{k})\) where \(k = \frac{1}{\text{hidden\_size}}\) Note For bidirectional GRUs, forward and backward are directions 0 and 1 respectively. Example of splitting the output layers when batch_first=False : output.view(seq_len, batch, num_directions, hidden_size) . Note batch_first argument is ignored for unbatched inputs. Examples: >>> rnn = tp . nn . GRU ( 10 , 20 , 2 ) >>> input = tp . randn ( 5 , 3 , 10 ) >>> h0 = tp . randn ( 2 , 3 , 20 ) >>> output , hn = rnn ( input , h0 ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module flatten_parameters ( ) → None Reset parameter data pointer so that they can use faster code paths. Right now, this works only if the module is on the GPU and cuDNN is enabled. Otherwise, it’s a no-op. TensorPlay has no cuDNN-backed fused RNN, so this is always a no-op here; the method is kept for torch API compatibility. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input , hx = None ) [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # GRUCell URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.rnn.GRUCell.html GRUCell class tensorplay.nn.modules.rnn. GRUCell ( input_size : int , hidden_size : int , bias : bool = True , device = None , dtype = None ) [source] A gated recurrent unit (GRU) cell. \[\begin{split}\begin{array}{ll} r = \sigma(W_{ir} x + b_{ir} + W_{hr} h + b_{hr}) \\ z = \sigma(W_{iz} x + b_{iz} + W_{hz} h + b_{hz}) \\ n = \tanh(W_{in} x + b_{in} + r \odot (W_{hn} h + b_{hn})) \\ h' = (1 - z) \odot n + z \odot h \end{array}\end{split}\] where \(\sigma\) is the sigmoid function, and \(\odot\) is the Hadamard product. Parameters : input_size – The number of expected features in the input x hidden_size – The number of features in the hidden state h bias – If False , then the layer does not use bias weights b_ih and b_hh . Default: True Inputs: input, hidden input : tensor containing input features hidden : tensor containing the initial hidden state for each element in the batch. Defaults to zero if not provided. Outputs: h’ h’ : tensor containing the next hidden state for each element in the batch Shape: input: \((N, H_{in})\) or \((H_{in})\) tensor containing input features where \(H_{in}\) = input_size . hidden: \((N, H_{out})\) or \((H_{out})\) tensor containing the initial hidden state where \(H_{out}\) = hidden_size . Defaults to zero if not provided. output: \((N, H_{out})\) or \((H_{out})\) tensor containing the next hidden state. weight_ih the learnable input-hidden weights, of shape (3*hidden_size, input_size) Type : tensorplay._C.TensorBase weight_hh the learnable hidden-hidden weights, of shape (3*hidden_size, hidden_size) Type : tensorplay._C.TensorBase bias_ih the learnable input-hidden bias, of shape (3*hidden_size) bias_hh the learnable hidden-hidden bias, of shape (3*hidden_size) Note All the weights and biases are initialized from \(\mathcal{U}(-\sqrt{k}, \sqrt{k})\) where \(k = \frac{1}{\text{hidden\_size}}\) Examples: >>> rnn = tp . nn . GRUCell ( 10 , 20 ) >>> input = tp . randn ( 6 , 3 , 10 ) >>> hx = tp . randn ( 3 , 20 ) >>> output = [] >>> for i in range ( 6 ): ... hx = rnn ( input [ i ], hx ) ... output . append ( hx ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # LSTM URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.rnn.LSTM.html LSTM class tensorplay.nn.modules.rnn. LSTM ( * args , ** kwargs ) [source] __init__(input_size, hidden_size, num_layers=1, bias=True, batch_first=False, dropout=0.0, bidirectional=False, proj_size=0, device=None, dtype=None) Apply a multi-layer long short-term memory (LSTM) RNN to an input sequence. For each element in the input sequence, each layer computes the following function: \[\begin{split}\begin{array}{ll} \\ i_t = \sigma(W_{ii} x_t + b_{ii} + W_{hi} h_{(t-1)} + b_{hi}) \\ f_t = \sigma(W_{if} x_t + b_{if} + W_{hf} h_{(t-1)} + b_{hf}) \\ g_t = \tanh(W_{ig} x_t + b_{ig} + W_{hg} h_{(t-1)} + b_{hg}) \\ o_t = \sigma(W_{io} x_t + b_{io} + W_{ho} h_{(t-1)} + b_{ho}) \\ c_t = f_t \odot c_{(t-1)} + i_t \odot g_t \\ h_t = o_t \odot \tanh(c_t) \\ \end{array}\end{split}\] where \(h_t\) is the hidden state at time t , \(c_t\) is the cell state at time t , \(x_t\) is the input at time t , \(h_{(t-1)}\) is the hidden state of the layer at time t-1 or the initial hidden state at time 0 , and \(i_t\) , \(f_t\) , \(g_t\) , \(o_t\) are the input, forget, cell, and output gates, respectively. \(\sigma\) is the sigmoid function, and \(\odot\) is the Hadamard product. In a multilayer LSTM, the input \(x^{(l)}_t\) of the \(l\) -th layer (:math`l >= 2`) is the hidden state \(h^{(l-1)}_t\) of the previous layer multiplied by dropout \(\varphi^{(l-1)}_t\) where each \(\varphi^{(l-1)}_t\) is a Bernoulli random variable which is \(0\) with probability dropout . If proj_size > 0 is specified, LSTM with projections will be used. This changes the LSTM cell in the following way. First, the dimension of \(h_t\) will be changed from hidden_size to proj_size (dimensions of \(W_{hi}\) will be changed accordingly). Second, the output hidden state of each layer will be multiplied by a learnable projection matrix: \(h_t = W_{hr}h_t\) . Note that as a consequence of this, the output of such LSTM will be of different shape as well. See Inputs/Outputs sections below for exact dimensions of all variables. Parameters : input_size – The number of expected features in the input x hidden_size – The number of features in the hidden state h num_layers – Number of recurrent layers. E.g., setting num_layers=2 would mean stacking two LSTMs together to form a stacked LSTM , with the second LSTM taking in outputs of the first LSTM and computing the final results. Default: 1 bias – If False , then the layer does not use bias weights b_ih and b_hh . Default: True batch_first – If True , then the input and output tensors are provided as (batch, seq, feature) instead of (seq, batch, feature) . Note that this does not apply to hidden or cell states. See the Inputs/Outputs sections below for details. Default: False dropout – If non-zero, introduces a Dropout layer on the outputs of each LSTM layer except the last layer, with dropout probability equal to dropout . Default: 0 bidirectional – If True , becomes a bidirectional LSTM. Default: False proj_size – If > 0 , will use LSTM with projections of corresponding size. Default: 0 Inputs: input, (h_0, c_0) input : tensor of shape \((L, H_{in})\) for unbatched input, \((L, N, H_{in})\) when batch_first=False or \((N, L, H_{in})\) when batch_first=True containing the features of the input sequence. The input can also be a packed variable length sequence. See tensorplay.nn.utils.rnn.pack_padded_sequence() for details. h_0 : tensor of shape \((D * \text{num\_layers}, H_{out})\) for unbatched input or \((D * \text{num\_layers}, N, H_{out})\) containing the initial hidden state for each element in the input sequence. Defaults to zeros if (h_0, c_0) is not provided. c_0 : tensor of shape \((D * \text{num\_layers}, H_{cell})\) for unbatched input or \((D * \text{num\_layers}, N, H_{cell})\) containing the initial cell state for each element in the input sequence. Defaults to zeros if (h_0, c_0) is not provided. where: \[\begin{split}\begin{aligned} N ={} & \text{batch size} \\ L ={} & \text{sequence length} \\ D ={} & 2 \text{ if bidirectional=True otherwise } 1 \\ H_{in} ={} & \text{input\_size} \\ H_{cell} ={} & \text{hidden\_size} \\ H_{out} ={} & \text{proj\_size if } \text{proj\_size}>0 \text{ otherwise hidden\_size} \\ \end{aligned}\end{split}\] Outputs: output, (h_n, c_n) output : tensor of shape \((L, D * H_{out})\) for unbatched input, \((L, N, D * H_{out})\) when batch_first=False or \((N, L, D * H_{out})\) when batch_first=True containing the output features (h_t) from the last layer of the LSTM, for each t . If a PackedSequence has been given as the input, the output will also be a packed sequence. h_n : tensor of shape \((D * \text{num\_layers}, H_{out})\) for unbatched input or \((D * \text{num\_layers}, N, H_{out})\) containing the final hidden state for each element in the sequence. c_n : tensor of shape \((D * \text{num\_layers}, H_{cell})\) for unbatched input or \((D * \text{num\_layers}, N, H_{cell})\) containing the final cell state for each element in the sequence. weight_ih_l [ k ] the learnable input-hidden weights of the k-th layer \((W_{ii}|W_{if}|W_{ig}|W_{io})\) , of shape (4*hidden_size, input_size) for k = 0 . Otherwise, the shape is (4*hidden_size, num_directions * hidden_size) . If proj_size > 0 was specified, the shape will be (4*hidden_size, num_directions * proj_size) . weight_hh_l [ k ] the learnable hidden-hidden weights of the k-th layer \((W_{hi}|W_{hf}|W_{hg}|W_{ho})\) , of shape (4*hidden_size, hidden_size) . If proj_size > 0 was specified, the shape will be (4*hidden_size, proj_size) . bias_ih_l [ k ] the learnable input-hidden bias of the k-th layer \((b_{ii}|b_{if}|b_{ig}|b_{io})\) , of shape (4*hidden_size) bias_hh_l [ k ] the learnable hidden-hidden bias of the k-th layer \((b_{hi}|b_{hf}|b_{hg}|b_{ho})\) , of shape (4*hidden_size) weight_hr_l [ k ] the learnable projection weights of the k-th layer \(W_{hr}\) , of shape (proj_size, hidden_size) . Only present when proj_size > 0 was specified. Note All the weights and biases are initialized from \(\mathcal{U}(-\sqrt{k}, \sqrt{k})\) where \(k = \frac{1}{\text{hidden\_size}}\) Note For bidirectional LSTMs, forward and backward are directions 0 and 1 respectively. Example of splitting the output layers when batch_first=False : output.view(seq_len, batch, num_directions, hidden_size) . Note batch_first argument is ignored for unbatched inputs. Note For unbatched input, the h_n and c_n are tensors of shape (num_layers * num_directions, H_out) and (num_layers * num_directions, H_cell) respectively. Examples: >>> rnn = tp . nn . LSTM ( 10 , 20 , 2 ) >>> input = tp . randn ( 5 , 3 , 10 ) >>> h0 = tp . randn ( 2 , 3 , 20 ) >>> c0 = tp . randn ( 2 , 3 , 20 ) >>> output , ( hn , cn ) = rnn ( input , ( h0 , c0 )) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module flatten_parameters ( ) → None Reset parameter data pointer so that they can use faster code paths. Right now, this works only if the module is on the GPU and cuDNN is enabled. Otherwise, it’s a no-op. TensorPlay has no cuDNN-backed fused RNN, so this is always a no-op here; the method is kept for torch API compatibility. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input , hx = None ) [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # LSTMCell URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.rnn.LSTMCell.html LSTMCell class tensorplay.nn.modules.rnn. LSTMCell ( input_size : int , hidden_size : int , bias : bool = True , device = None , dtype = None ) [source] A long short-term memory (LSTM) cell. \[\begin{split}\begin{array}{ll} i = \sigma(W_{ii} x + b_{ii} + W_{hi} h + b_{hi}) \\ f = \sigma(W_{if} x + b_{if} + W_{hf} h + b_{hf}) \\ g = \tanh(W_{ig} x + b_{ig} + W_{hg} h + b_{hg}) \\ o = \sigma(W_{io} x + b_{io} + W_{ho} h + b_{ho}) \\ c' = f \odot c + i \odot g \\ h' = o \odot \tanh(c') \\ \end{array}\end{split}\] where \(\sigma\) is the sigmoid function, and \(\odot\) is the Hadamard product. Parameters : input_size – The number of expected features in the input x hidden_size – The number of features in the hidden state h bias – If False , then the layer does not use bias weights b_ih and b_hh . Default: True Inputs: input, (h_0, c_0) input of shape (batch, input_size) or (input_size) : tensor containing input features h_0 of shape (batch, hidden_size) or (hidden_size) : tensor containing the initial hidden state c_0 of shape (batch, hidden_size) or (hidden_size) : tensor containing the initial cell state If (h_0, c_0) is not provided, both h_0 and c_0 default to zero. Outputs: (h_1, c_1) h_1 of shape (batch, hidden_size) or (hidden_size) : tensor containing the next hidden state c_1 of shape (batch, hidden_size) or (hidden_size) : tensor containing the next cell state weight_ih the learnable input-hidden weights, of shape (4*hidden_size, input_size) Type : tensorplay._C.TensorBase weight_hh the learnable hidden-hidden weights, of shape (4*hidden_size, hidden_size) Type : tensorplay._C.TensorBase bias_ih the learnable input-hidden bias, of shape (4*hidden_size) bias_hh the learnable hidden-hidden bias, of shape (4*hidden_size) Note All the weights and biases are initialized from \(\mathcal{U}(-\sqrt{k}, \sqrt{k})\) where \(k = \frac{1}{\text{hidden\_size}}\) Examples: >>> rnn = tp . nn . LSTMCell ( 10 , 20 ) # (input_size, hidden_size) >>> input = tp . randn ( 2 , 3 , 10 ) # (time_steps, batch, input_size) >>> hx = tp . randn ( 3 , 20 ) # (batch, hidden_size) >>> cx = tp . randn ( 3 , 20 ) >>> output = [] >>> for i in range ( input . size ()[ 0 ]): ... hx , cx = rnn ( input [ i ], ( hx , cx )) ... output . append ( hx ) >>> output = tp . stack ( output , dim = 0 ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # RNN URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.rnn.RNN.html RNN class tensorplay.nn.modules.rnn. RNN ( * args , ** kwargs ) [source] __init__(input_size, hidden_size, num_layers=1, nonlinearity=’tanh’, bias=True, batch_first=False, dropout=0.0, bidirectional=False, device=None, dtype=None) Apply a multi-layer Elman RNN with \(\tanh\) or \(\text{ReLU}\) non-linearity to an input sequence. For each element in the input sequence, each layer computes the following function: \[h_t = \tanh(x_t W_{ih}^T + b_{ih} + h_{t-1}W_{hh}^T + b_{hh})\] where \(h_t\) is the hidden state at time t , \(x_t\) is the input at time t , and \(h_{(t-1)}\) is the hidden state of the previous layer at time t-1 or the initial hidden state at time 0 . If nonlinearity is 'relu' , then \(\text{ReLU}\) is used instead of \(\tanh\) . Parameters : input_size – The number of expected features in the input x hidden_size – The number of features in the hidden state h num_layers – Number of recurrent layers. E.g., setting num_layers=2 would mean stacking two RNNs together to form a stacked RNN , with the second RNN taking in outputs of the first RNN and computing the final results. Default: 1 nonlinearity – The non-linearity to use. Can be either 'tanh' or 'relu' . Default: 'tanh' bias – If False , then the layer does not use bias weights b_ih and b_hh . Default: True batch_first – If True , then the input and output tensors are provided as (batch, seq, feature) instead of (seq, batch, feature) . Note that this does not apply to hidden or cell states. See the Inputs/Outputs sections below for details. Default: False dropout – If non-zero, introduces a Dropout layer on the outputs of each RNN layer except the last layer, with dropout probability equal to dropout . Default: 0 bidirectional – If True , becomes a bidirectional RNN. Default: False Inputs: input, hx input : tensor of shape \((L, H_{in})\) for unbatched input, \((L, N, H_{in})\) when batch_first=False or \((N, L, H_{in})\) when batch_first=True containing the features of the input sequence. The input can also be a packed variable length sequence. See tensorplay.nn.utils.rnn.pack_padded_sequence() or tensorplay.nn.utils.rnn.pack_sequence() for details. hx : tensor of shape \((D * \text{num\_layers}, H_{out})\) for unbatched input or \((D * \text{num\_layers}, N, H_{out})\) containing the initial hidden state for the input sequence batch. Defaults to zeros if not provided. where: \[\begin{split}\begin{aligned} N ={} & \text{batch size} \\ L ={} & \text{sequence length} \\ D ={} & 2 \text{ if bidirectional=True otherwise } 1 \\ H_{in} ={} & \text{input\_size} \\ H_{out} ={} & \text{hidden\_size} \end{aligned}\end{split}\] Outputs: output, h_n output : tensor of shape \((L, D * H_{out})\) for unbatched input, \((L, N, D * H_{out})\) when batch_first=False or \((N, L, D * H_{out})\) when batch_first=True containing the output features (h_t) from the last layer of the RNN, for each t . If a PackedSequence has been given as the input, the output will also be a packed sequence. h_n : tensor of shape \((D * \text{num\_layers}, H_{out})\) for unbatched input or \((D * \text{num\_layers}, N, H_{out})\) containing the final hidden state for each element in the batch. weight_ih_l [ k ] the learnable input-hidden weights of the k-th layer, of shape (hidden_size, input_size) for k = 0 . Otherwise, the shape is (hidden_size, num_directions * hidden_size) weight_hh_l [ k ] the learnable hidden-hidden weights of the k-th layer, of shape (hidden_size, hidden_size) bias_ih_l [ k ] the learnable input-hidden bias of the k-th layer, of shape (hidden_size) bias_hh_l [ k ] the learnable hidden-hidden bias of the k-th layer, of shape (hidden_size) Note All the weights and biases are initialized from \(\mathcal{U}(-\sqrt{k}, \sqrt{k})\) where \(k = \frac{1}{\text{hidden\_size}}\) Note For bidirectional RNNs, forward and backward are directions 0 and 1 respectively. Example of splitting the output layers when batch_first=False : output.view(seq_len, batch, num_directions, hidden_size) . Note batch_first argument is ignored for unbatched inputs. Examples: >>> rnn = tp . nn . RNN ( 10 , 20 , 2 ) >>> input = tp . randn ( 5 , 3 , 10 ) >>> h0 = tp . randn ( 2 , 3 , 20 ) >>> output , hn = rnn ( input , h0 ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module flatten_parameters ( ) → None Reset parameter data pointer so that they can use faster code paths. Right now, this works only if the module is on the GPU and cuDNN is enabled. Otherwise, it’s a no-op. TensorPlay has no cuDNN-backed fused RNN, so this is always a no-op here; the method is kept for torch API compatibility. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input , hx = None ) [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # RNNBase URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.rnn.RNNBase.html RNNBase class tensorplay.nn.modules.rnn. RNNBase ( mode : str , input_size : int , hidden_size : int , num_layers : int = 1 , bias : bool = True , batch_first : bool = False , dropout : float = 0.0 , bidirectional : bool = False , proj_size : int = 0 , device = None , dtype = None ) [source] Base class for RNN modules (RNN, LSTM, GRU). Implements aspects of RNNs shared by the RNN, LSTM, and GRU classes, such as module initialization and utility methods for parameter storage management. Note The forward method is not implemented by the RNNBase class. Note LSTM and GRU classes override some methods implemented by RNNBase. add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module flatten_parameters ( ) → None [source] Reset parameter data pointer so that they can use faster code paths. Right now, this works only if the module is on the GPU and cuDNN is enabled. Otherwise, it’s a no-op. TensorPlay has no cuDNN-backed fused RNN, so this is always a no-op here; the method is kept for torch API compatibility. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( * input : Any ) → None Define the computation performed at every call. Should be overridden by all subclasses. Note Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # RNNCell URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.rnn.RNNCell.html RNNCell class tensorplay.nn.modules.rnn. RNNCell ( input_size : int , hidden_size : int , bias : bool = True , nonlinearity : str = 'tanh' , device = None , dtype = None ) [source] An Elman RNN cell with tanh or ReLU non-linearity. \[h' = \tanh(W_{ih} x + b_{ih} + W_{hh} h + b_{hh})\] If nonlinearity is ‘relu’ , then ReLU is used in place of tanh. Parameters : input_size – The number of expected features in the input x hidden_size – The number of features in the hidden state h bias – If False , then the layer does not use bias weights b_ih and b_hh . Default: True nonlinearity – The non-linearity to use. Can be either 'tanh' or 'relu' . Default: 'tanh' Inputs: input, hidden input : tensor containing input features hidden : tensor containing the initial hidden state Defaults to zero if not provided. Outputs: h’ h’ of shape (batch, hidden_size) : tensor containing the next hidden state for each element in the batch Shape: input: \((N, H_{in})\) or \((H_{in})\) tensor containing input features where \(H_{in}\) = input_size . hidden: \((N, H_{out})\) or \((H_{out})\) tensor containing the initial hidden state where \(H_{out}\) = hidden_size . Defaults to zero if not provided. output: \((N, H_{out})\) or \((H_{out})\) tensor containing the next hidden state. weight_ih the learnable input-hidden weights, of shape (hidden_size, input_size) Type : tensorplay._C.TensorBase weight_hh the learnable hidden-hidden weights, of shape (hidden_size, hidden_size) Type : tensorplay._C.TensorBase bias_ih the learnable input-hidden bias, of shape (hidden_size) bias_hh the learnable hidden-hidden bias, of shape (hidden_size) Note All the weights and biases are initialized from \(\mathcal{U}(-\sqrt{k}, \sqrt{k})\) where \(k = \frac{1}{\text{hidden\_size}}\) Examples: >>> rnn = tp . nn . RNNCell ( 10 , 20 ) >>> input = tp . randn ( 6 , 3 , 10 ) >>> hx = tp . randn ( 3 , 20 ) >>> output = [] >>> for i in range ( 6 ): ... hx = rnn ( input [ i ], hx ) ... output . append ( hx ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Embedding URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.sparse.Embedding.html Embedding class tensorplay.nn.modules.sparse. Embedding ( num_embeddings : int , embedding_dim : int , padding_idx : int | None = None , max_norm : float | None = None , norm_type : float = 2.0 , scale_grad_by_freq : bool = False , sparse : bool = False , _weight : TensorBase | None = None , device = None , dtype = None ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # EmbeddingBag URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.sparse.EmbeddingBag.html EmbeddingBag class tensorplay.nn.modules.sparse. EmbeddingBag ( num_embeddings : int , embedding_dim : int , max_norm : float | None = None , norm_type : float = 2.0 , scale_grad_by_freq : bool = False , mode : str = 'mean' , sparse : bool = False , _weight : TensorBase | None = None , include_last_offset : bool = False , padding_idx : int | None = None , device = None , dtype = None ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module classmethod from_pretrained ( embeddings , freeze : bool = True , max_norm : float | None = None , norm_type : float = 2.0 , scale_grad_by_freq : bool = False , mode : str = 'mean' , sparse : bool = False , include_last_offset : bool = False , padding_idx : int | None = None ) → EmbeddingBag [source] Create EmbeddingBag instance from given 2-dimensional FloatTensor. Parameters : embeddings ( Tensor ) – FloatTensor containing weights for the EmbeddingBag. First dimension is being passed to EmbeddingBag as ‘num_embeddings’, second as ‘embedding_dim’. freeze ( bool , optional ) – If True , the tensor does not get updated in the learning process. Equivalent to embeddingbag.weight.requires_grad = False . Default: True get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Transformer URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.transformer.Transformer.html Transformer class tensorplay.nn.modules.transformer. Transformer ( d_model: int = 512, nhead: int = 8, num_encoder_layers: int = 6, num_decoder_layers: int = 6, dim_feedforward: int = 2048, dropout: float = 0.1, activation: str | ~collections.abc.Callable[[~tensorplay._C.TensorBase], ~tensorplay._C.TensorBase] = , custom_encoder: ~typing.Any | None = None, custom_decoder: ~typing.Any | None = None, layer_norm_eps: float = 1e-05, batch_first: bool = False, norm_first: bool = False, bias: bool = True, device=None, dtype=None ) [source] A basic transformer layer. This Transformer layer implements the original Transformer architecture described in the Attention Is All You Need paper. The intent of this layer is as a reference implementation for foundational understanding and thus it contains only limited features relative to newer Transformer architectures. Given the fast pace of innovation in transformer-like architectures, we recommend exploring this tutorial to build an efficient transformer layer from building blocks in core or using higher level libraries from the PyTorch Ecosystem . Parameters : d_model – the number of expected features in the encoder/decoder inputs (default=512). nhead – the number of heads in the multiheadattention models (default=8). num_encoder_layers – the number of sub-encoder-layers in the encoder (default=6). num_decoder_layers – the number of sub-decoder-layers in the decoder (default=6). dim_feedforward – the dimension of the feedforward network model (default=2048). dropout – the dropout value (default=0.1). activation – the activation function of encoder/decoder intermediate layer, can be a string (“relu” or “gelu”) or a unary callable. Default: relu custom_encoder – custom encoder (default=None). custom_decoder – custom decoder (default=None). layer_norm_eps – the eps value in layer normalization components (default=1e-5). batch_first – If True , then the input and output tensors are provided as (batch, seq, feature). Default: False (seq, batch, feature). norm_first – if True , encoder and decoder layers will perform LayerNorms before other attention and feedforward operations, otherwise after. Default: False (after). bias – If set to False , Linear and LayerNorm layers will not learn an additive bias. Default: True . Examples >>> transformer_model = nn . Transformer ( ... nhead = 16 , num_encoder_layers = 12 , batch_first = True ... ) >>> src = torch . rand (( 32 , 10 , 512 )) >>> tgt = torch . rand (( 32 , 20 , 512 )) >>> out = transformer_model ( src , tgt ) Note: A full example to apply nn.Transformer module for the word language model is available in pytorch/examples add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( src : TensorBase , tgt : TensorBase , src_mask : TensorBase | None = None , tgt_mask : TensorBase | None = None , memory_mask : TensorBase | None = None , src_key_padding_mask : TensorBase | None = None , tgt_key_padding_mask : TensorBase | None = None , memory_key_padding_mask : TensorBase | None = None , src_is_causal : bool | None = None , tgt_is_causal : bool | None = None , memory_is_causal : bool = False ) → TensorBase [source] Take in and process masked source/target sequences. Note If a boolean tensor is provided for any of the [src/tgt/memory]_mask arguments, positions with a True value are not allowed to participate in the attention, which is the opposite of the definition for attn_mask in torch.nn.functional.scaled_dot_product_attention() . Parameters : src – the sequence to the encoder (required). tgt – the sequence to the decoder (required). src_mask – the additive mask for the src sequence (optional). tgt_mask – the additive mask for the tgt sequence (optional). memory_mask – the additive mask for the encoder output (optional). src_key_padding_mask – the Tensor mask for src keys per batch (optional). tgt_key_padding_mask – the Tensor mask for tgt keys per batch (optional). memory_key_padding_mask – the Tensor mask for memory keys per batch (optional). src_is_causal – If specified, applies a causal mask as src_mask . Default: None ; try to detect a causal mask. Warning: src_is_causal provides a hint that src_mask is the causal mask. Providing incorrect hints can result in incorrect execution, including forward and backward compatibility. tgt_is_causal – If specified, applies a causal mask as tgt_mask . Default: None ; try to detect a causal mask. Warning: tgt_is_causal provides a hint that tgt_mask is the causal mask. Providing incorrect hints can result in incorrect execution, including forward and backward compatibility. memory_is_causal – If specified, applies a causal mask as memory_mask . Default: False . Warning: memory_is_causal provides a hint that memory_mask is the causal mask. Providing incorrect hints can result in incorrect execution, including forward and backward compatibility. Shape: src: \((S, E)\) for unbatched input, \((S, N, E)\) if batch_first=False or (N, S, E) if batch_first=True . tgt: \((T, E)\) for unbatched input, \((T, N, E)\) if batch_first=False or (N, T, E) if batch_first=True . src_mask: \((S, S)\) or \((N\cdot\text{num\_heads}, S, S)\) . tgt_mask: \((T, T)\) or \((N\cdot\text{num\_heads}, T, T)\) . memory_mask: \((T, S)\) . src_key_padding_mask: \((S)\) for unbatched input otherwise \((N, S)\) . tgt_key_padding_mask: \((T)\) for unbatched input otherwise \((N, T)\) . memory_key_padding_mask: \((S)\) for unbatched input otherwise \((N, S)\) . Note: [src/tgt/memory]_mask ensures that position \(i\) is allowed to attend the unmasked positions. If a BoolTensor is provided, positions with True are not allowed to attend while False values will be unchanged. If a FloatTensor is provided, it will be added to the attention weight. [src/tgt/memory]_key_padding_mask provides specified elements in the key to be ignored by the attention. If a BoolTensor is provided, the positions with the value of True will be ignored while the position with the value of False will be unchanged. output: \((T, E)\) for unbatched input, \((T, N, E)\) if batch_first=False or (N, T, E) if batch_first=True . Note: Due to the multi-head attention architecture in the transformer model, the output sequence length of a transformer is same as the input sequence (i.e. target) length of the decoder. where \(S\) is the source sequence length, \(T\) is the target sequence length, \(N\) is the batch size, \(E\) is the feature number Examples >>> # xdoctest: +SKIP >>> output = transformer_model ( ... src , tgt , src_mask = src_mask , tgt_mask = tgt_mask ... ) static generate_square_subsequent_mask ( sz : int , device = None , dtype = None ) → TensorBase [source] Generate a square causal mask for the sequence. The masked positions are filled with float(‘-inf’). Unmasked positions are filled with float(0.0). get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # TransformerDecoder URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.transformer.TransformerDecoder.html TransformerDecoder class tensorplay.nn.modules.transformer. TransformerDecoder ( decoder_layer : TransformerDecoderLayer , num_layers : int , norm : Module | None = None ) [source] TransformerDecoder is a stack of N decoder layers. This TransformerDecoder layer implements the original architecture described in the Attention Is All You Need paper. The intent of this layer is as a reference implementation for foundational understanding and thus it contains only limited features relative to newer Transformer architectures. Given the fast pace of innovation in transformer-like architectures, we recommend exploring this tutorial to build efficient layers from building blocks in core or using higher level libraries from the PyTorch Ecosystem . Warning All layers in the TransformerDecoder are initialized with the same parameters. It is recommended to manually initialize the layers after creating the TransformerDecoder instance. Parameters : decoder_layer – an instance of the TransformerDecoderLayer() class (required). num_layers – the number of sub-decoder-layers in the decoder (required). norm – the layer normalization component (optional). Examples >>> decoder_layer = nn . TransformerDecoderLayer ( d_model = 512 , nhead = 8 ) >>> transformer_decoder = nn . TransformerDecoder ( decoder_layer , num_layers = 6 ) >>> memory = torch . rand ( 10 , 32 , 512 ) >>> tgt = torch . rand ( 20 , 32 , 512 ) >>> out = transformer_decoder ( tgt , memory ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( tgt : TensorBase , memory : TensorBase , tgt_mask : TensorBase | None = None , memory_mask : TensorBase | None = None , tgt_key_padding_mask : TensorBase | None = None , memory_key_padding_mask : TensorBase | None = None , tgt_is_causal : bool | None = None , memory_is_causal : bool = False ) → TensorBase [source] Pass the inputs (and mask) through the decoder layer in turn. Parameters : tgt – the sequence to the decoder (required). memory – the sequence from the last layer of the encoder (required). tgt_mask – the mask for the tgt sequence (optional). memory_mask – the mask for the memory sequence (optional). tgt_key_padding_mask – the mask for the tgt keys per batch (optional). memory_key_padding_mask – the mask for the memory keys per batch (optional). tgt_is_causal – If specified, applies a causal mask as tgt mask . Default: None ; try to detect a causal mask. Warning: tgt_is_causal provides a hint that tgt_mask is the causal mask. Providing incorrect hints can result in incorrect execution, including forward and backward compatibility. memory_is_causal – If specified, applies a causal mask as memory mask . Default: False . Warning: memory_is_causal provides a hint that memory_mask is the causal mask. Providing incorrect hints can result in incorrect execution, including forward and backward compatibility. Shape: see the docs in Transformer . get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # TransformerDecoderLayer URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.transformer.TransformerDecoderLayer.html TransformerDecoderLayer class tensorplay.nn.modules.transformer. TransformerDecoderLayer ( d_model: int, nhead: int, dim_feedforward: int = 2048, dropout: float = 0.1, activation: str | ~collections.abc.Callable[[~tensorplay._C.TensorBase], ~tensorplay._C.TensorBase] = , layer_norm_eps: float = 1e-05, batch_first: bool = False, norm_first: bool = False, bias: bool = True, device=None, dtype=None ) [source] TransformerDecoderLayer is made up of self-attn, multi-head-attn and feedforward network. This TransformerDecoderLayer implements the original architecture described in the Attention Is All You Need paper. The intent of this layer is as a reference implementation for foundational understanding and thus it contains only limited features relative to newer Transformer architectures. Given the fast pace of innovation in transformer-like architectures, we recommend exploring this tutorial to build efficient layers from building blocks in core or using higher level libraries from the PyTorch Ecosystem . Parameters : d_model – the number of expected features in the input (required). nhead – the number of heads in the multiheadattention models (required). dim_feedforward – the dimension of the feedforward network model (default=2048). dropout – the dropout value (default=0.1). activation – the activation function of the intermediate layer, can be a string (“relu” or “gelu”) or a unary callable. Default: relu layer_norm_eps – the eps value in layer normalization components (default=1e-5). batch_first – If True , then the input and output tensors are provided as (batch, seq, feature). Default: False (seq, batch, feature). norm_first – if True , layer norm is done prior to self attention, multihead attention and feedforward operations, respectively. Otherwise it’s done after. Default: False (after). bias – If set to False , Linear and LayerNorm layers will not learn an additive bias. Default: True . Examples >>> decoder_layer = nn . TransformerDecoderLayer ( d_model = 512 , nhead = 8 ) >>> memory = torch . rand ( 10 , 32 , 512 ) >>> tgt = torch . rand ( 20 , 32 , 512 ) >>> out = decoder_layer ( tgt , memory ) Alternatively, when batch_first is True : >>> decoder_layer = nn . TransformerDecoderLayer ( ... d_model = 512 , nhead = 8 , batch_first = True ... ) >>> memory = torch . rand ( 32 , 10 , 512 ) >>> tgt = torch . rand ( 32 , 20 , 512 ) >>> out = decoder_layer ( tgt , memory ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( tgt : TensorBase , memory : TensorBase , tgt_mask : TensorBase | None = None , memory_mask : TensorBase | None = None , tgt_key_padding_mask : TensorBase | None = None , memory_key_padding_mask : TensorBase | None = None , tgt_is_causal : bool = False , memory_is_causal : bool = False ) → TensorBase [source] Pass the inputs (and mask) through the decoder layer. Parameters : tgt – the sequence to the decoder layer (required). memory – the sequence from the last layer of the encoder (required). tgt_mask – the mask for the tgt sequence (optional). memory_mask – the mask for the memory sequence (optional). tgt_key_padding_mask – the mask for the tgt keys per batch (optional). memory_key_padding_mask – the mask for the memory keys per batch (optional). tgt_is_causal – If specified, applies a causal mask as tgt mask . Default: False . Warning: tgt_is_causal provides a hint that tgt_mask is the causal mask. Providing incorrect hints can result in incorrect execution, including forward and backward compatibility. memory_is_causal – If specified, applies a causal mask as memory mask . Default: False . Warning: memory_is_causal provides a hint that memory_mask is the causal mask. Providing incorrect hints can result in incorrect execution, including forward and backward compatibility. Shape: see the docs in Transformer . get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # TransformerEncoder URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.transformer.TransformerEncoder.html TransformerEncoder class tensorplay.nn.modules.transformer. TransformerEncoder ( encoder_layer : TransformerEncoderLayer , num_layers : int , norm : Module | None = None , enable_nested_tensor : bool = True , mask_check : bool = True ) [source] TransformerEncoder is a stack of N encoder layers. This TransformerEncoder layer implements the original architecture described in the Attention Is All You Need paper. The intent of this layer is as a reference implementation for foundational understanding and thus it contains only limited features relative to newer Transformer architectures. Given the fast pace of innovation in transformer-like architectures, we recommend exploring this tutorial to build efficient layers from building blocks in core or using higher level libraries from the PyTorch Ecosystem . Warning All layers in the TransformerEncoder are initialized with the same parameters. It is recommended to manually initialize the layers after creating the TransformerEncoder instance. Parameters : encoder_layer – an instance of the TransformerEncoderLayer() class (required). num_layers – the number of sub-encoder-layers in the encoder (required). norm – the layer normalization component (optional). enable_nested_tensor – if True, input will automatically convert to nested tensor (and convert back on output). This will improve the overall performance of TransformerEncoder when padding rate is high. Default: True (enabled). Examples >>> encoder_layer = nn . TransformerEncoderLayer ( ... d_model = 512 , nhead = 8 , batch_first = True ... ) >>> transformer_encoder = nn . TransformerEncoder ( encoder_layer , num_layers = 6 ) >>> src = torch . rand ( 32 , 10 , 512 ) >>> out = transformer_encoder ( src ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( src : TensorBase , mask : TensorBase | None = None , src_key_padding_mask : TensorBase | None = None , is_causal : bool | None = None ) → TensorBase [source] Pass the input through the encoder layers in turn. Parameters : src – the sequence to the encoder (required). mask – the mask for the src sequence (optional). src_key_padding_mask – the mask for the src keys per batch (optional). is_causal – If specified, applies a causal mask as mask . Default: None ; try to detect a causal mask. Warning: is_causal provides a hint that mask is the causal mask. Providing incorrect hints can result in incorrect execution, including forward and backward compatibility. Shape: see the docs in Transformer . get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # TransformerEncoderLayer URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.transformer.TransformerEncoderLayer.html TransformerEncoderLayer class tensorplay.nn.modules.transformer. TransformerEncoderLayer ( d_model: int, nhead: int, dim_feedforward: int = 2048, dropout: float = 0.1, activation: str | ~collections.abc.Callable[[~tensorplay._C.TensorBase], ~tensorplay._C.TensorBase] = , layer_norm_eps: float = 1e-05, batch_first: bool = False, norm_first: bool = False, bias: bool = True, device=None, dtype=None ) [source] TransformerEncoderLayer is made up of self-attn and feedforward network. This TransformerEncoderLayer implements the original architecture described in the Attention Is All You Need paper. The intent of this layer is as a reference implementation for foundational understanding and thus it contains only limited features relative to newer Transformer architectures. Given the fast pace of innovation in transformer-like architectures, we recommend exploring this tutorial to build efficient layers from building blocks in core or using higher level libraries from the PyTorch Ecosystem . Parameters : d_model – the number of expected features in the input (required). nhead – the number of heads in the multiheadattention models (required). dim_feedforward – the dimension of the feedforward network model (default=2048). dropout – the dropout value (default=0.1). activation – the activation function of the intermediate layer, can be a string (“relu” or “gelu”) or a unary callable. Default: relu layer_norm_eps – the eps value in layer normalization components (default=1e-5). batch_first – If True , then the input and output tensors are provided as (batch, seq, feature). Default: False (seq, batch, feature). norm_first – if True , layer norm is done prior to attention and feedforward operations, respectively. Otherwise it’s done after. Default: False (after). bias – If set to False , Linear and LayerNorm layers will not learn an additive bias. Default: True . Examples >>> encoder_layer = nn . TransformerEncoderLayer ( d_model = 512 , nhead = 8 ) >>> src = torch . rand ( 10 , 32 , 512 ) >>> out = encoder_layer ( src ) Alternatively, when batch_first is True : >>> encoder_layer = nn . TransformerEncoderLayer ( ... d_model = 512 , nhead = 8 , batch_first = True ... ) >>> src = torch . rand ( 32 , 10 , 512 ) >>> out = encoder_layer ( src ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( src : TensorBase , src_mask : TensorBase | None = None , src_key_padding_mask : TensorBase | None = None , is_causal : bool = False ) → TensorBase [source] Pass the input through the encoder layer. Parameters : src – the sequence to the encoder layer (required). src_mask – the mask for the src sequence (optional). src_key_padding_mask – the mask for the src keys per batch (optional). is_causal – If specified, applies a causal mask as src mask . Default: False . Warning: is_causal provides a hint that src_mask is the causal mask. Providing incorrect hints can result in incorrect execution, including forward and backward compatibility. Shape: see the docs in Transformer . get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Upsample URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.upsampling.Upsample.html Upsample class tensorplay.nn.modules.upsampling. Upsample ( size : int | tuple [ int , ... ] | None = None , scale_factor : float | tuple [ float , ... ] | None = None , mode : str = 'nearest' , align_corners : bool | None = None , recompute_scale_factor : bool | None = None ) [source] Upsamples a given multi-channel 1D (temporal), 2D (spatial) or 3D (volumetric) data. The input data is assumed to be of the form minibatch x channels x [optional depth] x [optional height] x width . Hence, for spatial inputs, we expect a 4D Tensor and for volumetric inputs, we expect a 5D Tensor. The algorithms available for upsampling are nearest neighbor and linear, bilinear, bicubic and trilinear for 3D, 4D and 5D input Tensor, respectively. One can either give a scale_factor or the target output size to calculate the output size. (You cannot give both, as it is ambiguous) Parameters : size ( int or Tuple [ int ] or Tuple [ int , int ] or Tuple [ int , int , int ] , optional ) – output spatial sizes scale_factor ( float or Tuple [ float ] or Tuple [ float , float ] or Tuple [ float , float , float ] , optional ) – multiplier for spatial size. Has to match input size if it is a tuple. mode ( str , optional ) – the upsampling algorithm: one of 'nearest' , 'linear' , 'bilinear' , 'bicubic' and 'trilinear' . Default: 'nearest' align_corners ( bool , optional ) – if True , the corner pixels of the input and output tensors are aligned, and thus preserving the values at those pixels. This only has effect when mode is 'linear' , 'bilinear' , 'bicubic' , or 'trilinear' . Default: False recompute_scale_factor ( bool , optional ) – recompute the scale_factor for use in the interpolation calculation. If recompute_scale_factor is True , then scale_factor must be passed in and scale_factor is used to compute the output size . The computed output size will be used to infer new scales for the interpolation. Note that when scale_factor is floating-point, it may differ from the recomputed scale_factor due to rounding and precision issues. If recompute_scale_factor is False , then size or scale_factor will be used directly for interpolation. Shape: Input: \((N, C, W_{in})\) , \((N, C, H_{in}, W_{in})\) or \((N, C, D_{in}, H_{in}, W_{in})\) Output: \((N, C, W_{out})\) , \((N, C, H_{out}, W_{out})\) or \((N, C, D_{out}, H_{out}, W_{out})\) , where \[D_{out} = \left\lfloor D_{in} \times \text{scale\_factor} \right\rfloor\] \[H_{out} = \left\lfloor H_{in} \times \text{scale\_factor} \right\rfloor\] \[W_{out} = \left\lfloor W_{in} \times \text{scale\_factor} \right\rfloor\] Warning With align_corners = True , the linearly interpolating modes ( linear , bilinear , bicubic , and trilinear ) don’t proportionally align the output and input pixels, and thus the output values can depend on the input size. This was the default behavior for these modes up to version 0.3.1. Since then, the default behavior is align_corners = False . See below for concrete examples on how this affects the outputs. Note If you want downsampling/general resizing, you should use interpolate() . Examples: >>> input = torch . arange ( 1 , 5 , dtype = torch . float32 ) . view ( 1 , 1 , 2 , 2 ) >>> input tensor([[[[1., 2.], [3., 4.]]]]) >>> m = nn . Upsample ( scale_factor = 2 , mode = 'nearest' ) >>> m ( input ) tensor([[[[1., 1., 2., 2.], [1., 1., 2., 2.], [3., 3., 4., 4.], [3., 3., 4., 4.]]]]) >>> # xdoctest: +IGNORE_WANT("other tests seem to modify printing styles") >>> m = nn . Upsample ( scale_factor = 2 , mode = 'bilinear' ) # align_corners=False >>> m ( input ) tensor([[[[1.0000, 1.2500, 1.7500, 2.0000], [1.5000, 1.7500, 2.2500, 2.5000], [2.5000, 2.7500, 3.2500, 3.5000], [3.0000, 3.2500, 3.7500, 4.0000]]]]) >>> m = nn . Upsample ( scale_factor = 2 , mode = 'bilinear' , align_corners = True ) >>> m ( input ) tensor([[[[1.0000, 1.3333, 1.6667, 2.0000], [1.6667, 2.0000, 2.3333, 2.6667], [2.3333, 2.6667, 3.0000, 3.3333], [3.0000, 3.3333, 3.6667, 4.0000]]]]) >>> # Try scaling the same data in a larger tensor >>> input_3x3 = torch . zeros ( 3 , 3 ) . view ( 1 , 1 , 3 , 3 ) >>> input_3x3 [:, :, : 2 , : 2 ] . copy_ ( input ) tensor([[[[1., 2.], [3., 4.]]]]) >>> input_3x3 tensor([[[[1., 2., 0.], [3., 4., 0.], [0., 0., 0.]]]]) >>> # xdoctest: +IGNORE_WANT("seems to fail when other tests are run in the same session") >>> m = nn . Upsample ( scale_factor = 2 , mode = 'bilinear' ) # align_corners=False >>> # Notice that values in top left corner are the same with the small input (except at boundary) >>> m ( input_3x3 ) tensor([[[[1.0000, 1.2500, 1.7500, 1.5000, 0.5000, 0.0000], [1.5000, 1.7500, 2.2500, 1.8750, 0.6250, 0.0000], [2.5000, 2.7500, 3.2500, 2.6250, 0.8750, 0.0000], [2.2500, 2.4375, 2.8125, 2.2500, 0.7500, 0.0000], [0.7500, 0.8125, 0.9375, 0.7500, 0.2500, 0.0000], [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000]]]]) >>> m = nn . Upsample ( scale_factor = 2 , mode = 'bilinear' , align_corners = True ) >>> # Notice that values in top left corner are now changed >>> m ( input_3x3 ) tensor([[[[1.0000, 1.4000, 1.8000, 1.6000, 0.8000, 0.0000], [1.8000, 2.2000, 2.6000, 2.2400, 1.1200, 0.0000], [2.6000, 3.0000, 3.4000, 2.8800, 1.4400, 0.0000], [2.4000, 2.7200, 3.0400, 2.5600, 1.2800, 0.0000], [1.2000, 1.3600, 1.5200, 1.2800, 0.6400, 0.0000], [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000]]]]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str [source] Return the extra representation of the module. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase [source] Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # UpsamplingBilinear2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.upsampling.UpsamplingBilinear2d.html UpsamplingBilinear2d class tensorplay.nn.modules.upsampling. UpsamplingBilinear2d ( size : int | tuple [ int , int ] | None = None , scale_factor : float | tuple [ float , float ] | None = None ) [source] Applies a 2D bilinear upsampling to an input signal composed of several input channels. To specify the scale, it takes either the size or the scale_factor as its constructor argument. When size is given, it is the output size of the image (h, w) . Parameters : size ( int or Tuple [ int , int ] , optional ) – output spatial sizes scale_factor ( float or Tuple [ float , float ] , optional ) – multiplier for spatial size. Warning This class is deprecated in favor of interpolate() . It is equivalent to nn.functional.interpolate(..., mode='bilinear', align_corners=True) . Shape: Input: \((N, C, H_{in}, W_{in})\) Output: \((N, C, H_{out}, W_{out})\) where \[H_{out} = \left\lfloor H_{in} \times \text{scale\_factor} \right\rfloor\] \[W_{out} = \left\lfloor W_{in} \times \text{scale\_factor} \right\rfloor\] Examples: >>> input = torch . arange ( 1 , 5 , dtype = torch . float32 ) . view ( 1 , 1 , 2 , 2 ) >>> input tensor([[[[1., 2.], [3., 4.]]]]) >>> # xdoctest: +IGNORE_WANT("do other tests modify the global state?") >>> m = nn . UpsamplingBilinear2d ( scale_factor = 2 ) >>> m ( input ) tensor([[[[1.0000, 1.3333, 1.6667, 2.0000], [1.6667, 2.0000, 2.3333, 2.6667], [2.3333, 2.6667, 3.0000, 3.3333], [3.0000, 3.3333, 3.6667, 4.0000]]]]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # UpsamplingNearest2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.modules.upsampling.UpsamplingNearest2d.html UpsamplingNearest2d class tensorplay.nn.modules.upsampling. UpsamplingNearest2d ( size : int | tuple [ int , int ] | None = None , scale_factor : float | tuple [ float , float ] | None = None ) [source] Applies a 2D nearest neighbor upsampling to an input signal composed of several input channels. To specify the scale, it takes either the size or the scale_factor as its constructor argument. When size is given, it is the output size of the image (h, w) . Parameters : size ( int or Tuple [ int , int ] , optional ) – output spatial sizes scale_factor ( float or Tuple [ float , float ] , optional ) – multiplier for spatial size. Warning This class is deprecated in favor of interpolate() . Shape: Input: \((N, C, H_{in}, W_{in})\) Output: \((N, C, H_{out}, W_{out})\) where \[H_{out} = \left\lfloor H_{in} \times \text{scale\_factor} \right\rfloor\] \[W_{out} = \left\lfloor W_{in} \times \text{scale\_factor} \right\rfloor\] Examples: >>> input = torch . arange ( 1 , 5 , dtype = torch . float32 ) . view ( 1 , 1 , 2 , 2 ) >>> input tensor([[[[1., 2.], [3., 4.]]]]) >>> m = nn . UpsamplingNearest2d ( scale_factor = 2 ) >>> m ( input ) tensor([[[[1., 1., 2., 2.], [1., 1., 2., 2.], [3., 3., 4., 4.], [3., 3., 4., 4.]]]]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # NonDynamicallyQuantizableLinear URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.NonDynamicallyQuantizableLinear.html NonDynamicallyQuantizableLinear class tensorplay.nn. NonDynamicallyQuantizableLinear ( * args , ** kwargs ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( input : TensorBase ) → TensorBase Runs the forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module reset_parameters ( ) → None Resets parameters based on their initialization used in __init__ . set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # DataParallel URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.parallel.data_parallel.DataParallel.html DataParallel class tensorplay.nn.parallel.data_parallel. DataParallel ( module , device_ids = None , output_device = None , dim = 0 ) [source] Implements data parallelism at the module level. This container parallelizes the application of the given module by splitting the input along the batch dimension (dimension 0). In the forward pass, the module is replicated on each device, each replica handles a slice of the input, and outputs are gathered on output_device . The batch size should be larger than the number of GPUs used. It is also recommended to use lr linearly scaled with the effective world size. Parameters : module – module to be parallelized device_ids ( int or torch.device or list ) – CUDA devices (default: all devices) output_device ( int or torch.device ) – device location of output (default: device_ids[0]) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # DistributedDataParallel URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.parallel.distributed.DistributedDataParallel.html DistributedDataParallel class tensorplay.nn.parallel.distributed. DistributedDataParallel ( module , device_ids = None , output_device = None , dim = 0 , broadcast_buffers = True , process_group = None , bucket_cap_mb = None , find_unused_parameters = False , check_reduction = False , gradient_as_bucket_view = False , static_graph = False ) [source] Implements distributed data parallelism (torch parity). Arguments mirror torch.nn.parallel.DistributedDataParallel . Gradient reduction uses buckets built by dist._compute_bucket_assignment_by_size ; each parameter’s post accumulate hook copies its gradient into its bucket buffer and, when the bucket completes, one all-reduce averages it and the reduced values are copied back into param.grad . Example: >>> tp . distributed . init_process_group ( backend = "nccl" ) >>> net = DistributedDataParallel ( model , device_ids = [ rank ]) >>> out = net ( input ) add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module join ( divide_by_initial_world_size : bool = True , enable : bool = True , throw_on_early_termination : bool = False ) [source] Context manager for training with uneven inputs across processes in DDP. This context manager will keep track of already-joined DDP processes, and “shadow” the forward and backward passes by inserting collective communication operations to match with the ones created by non-joined DDP processes. See torch’s Join docs for details. join_hook ( ** kwargs ) [source] DDP join hook enables training on uneven inputs by mirroring communications in forward and backward passes. Parameters : kwargs ( dict ) – a dict containing any keyword arguments to modify the behavior of the join hook at run time; all Joinable instances sharing the same join context manager are forwarded the same value for kwargs . The hook supports the following keyword arguments: divide_by_initial_world_size (bool, optional): If True , then gradients are divided by the initial world size that DDP was launched with. Default is True . load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_comm_hook ( state : object , hook ) → None [source] Register communication hook for custom gradient aggregation. The hook has signature hook(state, bucket) -> Future[Tensor] where the future holds the reduced bucket buffer (torch contract; completion is resolved synchronously on wait ). register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # Parameter URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.Parameter.html Parameter class tensorplay.nn. Parameter ( data = None , requires_grad = True ) [source] A kind of Tensor that is to be considered a module parameter. Parameters are Tensor subclasses, that have a very special property when used with Module s - when they’re assigned as Module attributes they are automatically added to the list of its parameters, and will appear e.g. in parameters() iterator. Assigning a Tensor doesn’t have such effect. This is because one might want to cache some temporary state, like last hidden state of the RNN, in the model. If there was no such class as Parameter , these temporaries would get registered too. Parameters : data ( Tensor ) – parameter tensor. requires_grad ( bool , optional ) – if the parameter requires gradient. Note that the tensorplay.no_grad() context does NOT affect the default behavior of Parameter creation–the Parameter will always have requires_grad=True unless given explicitly. Default: True abs ( Tensor self ) → Tensor abs_ ( Tensor(a!) self) -> Tensor(a! ) acos ( Tensor self ) → Tensor acosh ( Tensor self ) → Tensor add ( ) add_ ( ) add_ .Tensor(Tensor(a!) self, Tensor other, * , Scalar alpha=1) -> Tensor(a!) | add_ .Scalar(Tensor(a!) self, Scalar other, Scalar alpha=1) -> Tensor(a!) addbmm ( Tensor self , Tensor batch1 , Tensor batch2 , * , Scalar beta=1 , Scalar alpha=1 ) → Tensor addcdiv ( Tensor self , Tensor tensor1 , Tensor tensor2 , * , Scalar value=1 ) → Tensor addcdiv_ ( Tensor(a!) self , Tensor tensor1 , Tensor tensor2 , * , Scalar value=1) -> Tensor(a! ) addcmul ( Tensor self , Tensor tensor1 , Tensor tensor2 , * , Scalar value=1 ) → Tensor addcmul_ ( Tensor(a!) self , Tensor tensor1 , Tensor tensor2 , * , Scalar value=1) -> Tensor(a! ) addmv ( Tensor self , Tensor mat , Tensor vec , * , Scalar beta=1 , Scalar alpha=1 ) → Tensor addr ( Tensor self , Tensor vec1 , Tensor vec2 , * , Scalar beta=1 , Scalar alpha=1 ) → Tensor airy_ai ( Tensor self ) → Tensor all ( Tensor self) -> Tensor | all.dim(Tensor self , int[] dim , bool keepdim=false ) → Tensor allclose ( Tensor self , Tensor other , float rtol=1e-05 , float atol=1e-08 , bool equal_nan=False ) → bool angle ( Tensor self ) → Tensor any ( Tensor self) -> Tensor | any.dim(Tensor self , int[] dim , bool keepdim=false ) → Tensor argmax ( Tensor self , int? dim=None , bool keepdim=False ) → Tensor argmin ( Tensor self , int? dim=None , bool keepdim=False ) → Tensor argsort ( Tensor self , int dim=-1 , bool descending=False ) → Tensor argwhere ( Tensor self ) → Tensor as_strided ( self : tensorplay._C.TensorBase , size : collections.abc.Sequence [ SupportsInt | SupportsIndex ] , stride : collections.abc.Sequence [ SupportsInt | SupportsIndex ] , storage_offset : SupportsInt | SupportsIndex | None = None ) → tensorplay._C.TensorBase asin ( Tensor self ) → Tensor asinh ( Tensor self ) → Tensor atan ( Tensor self ) → Tensor atan2 ( Tensor self , Tensor other ) → Tensor atanh ( Tensor self ) → Tensor backward ( self : tensorplay._C.TensorBase , gradient : tensorplay._C.TensorBase | None = None , retain_graph : bool | None = None , create_graph : bool = False ) → None bernoulli ( Tensor self ) → Tensor bernoulli_ ( Tensor(a!) self) -> Tensor(a! ) bessel_j0 ( Tensor self ) → Tensor bessel_j1 ( Tensor self ) → Tensor bessel_y0 ( Tensor self ) → Tensor bessel_y1 ( Tensor self ) → Tensor bincount ( Tensor self , Tensor? weights=None , int minlength=0 ) → Tensor bitwise_and ( ) bitwise_left_shift ( ) bitwise_not ( Tensor self ) → Tensor bitwise_or ( ) bitwise_right_shift ( ) bitwise_xor ( ) bmm ( Tensor self , Tensor mat2 ) → Tensor broadcast_to ( Tensor self , int[] size ) → Tensor cauchy_ ( Tensor(a!) self , float median=0.0 , float sigma=1.0) -> Tensor(a! ) ceil ( Tensor self ) → Tensor celu ( Tensor self , Scalar alpha=1.0 ) → Tensor channel_shuffle ( Tensor self , int groups ) → Tensor cholesky ( Tensor self , bool upper=False ) → Tensor cholesky_inverse ( Tensor self , bool upper=False ) → Tensor cholesky_solve ( Tensor self , Tensor input2 , bool upper=False ) → Tensor chunk ( Tensor self , int chunks , int dim=0 ) → Tensor[] clamp ( Tensor self , Scalar? min=None , Scalar? max=None ) → Tensor clamp_ ( Tensor(a!) self , Scalar? min=None , Scalar? max=None) -> Tensor(a! ) clip ( Tensor self , Scalar? min=None , Scalar? max=None ) → Tensor clone ( Tensor self , * , MemoryFormat? memory_format=None ) → Tensor coalesce ( self : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase col_indices ( self : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase conj ( Tensor self ) → Tensor contiguous ( Tensor self , * , MemoryFormat memory_format=Contiguous ) → Tensor copy_ ( Tensor(a!) self , Tensor src , bool non_blocking=False) -> Tensor(a! ) cos ( Tensor self ) → Tensor cosh ( Tensor self ) → Tensor cpu ( ) Returns a copy of this object in CPU memory. If this object is already in CPU memory, then no copy is performed and the original object is returned. crow_indices ( self : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase cuda ( device = None , non_blocking = False ) Returns a copy of this object in CUDA memory. If this object is already in CUDA memory and on the correct device, then no copy is performed and the original object is returned. cummax ( Tensor self , int dim) -> (Tensor values , Tensor indices ) cummin ( Tensor self , int dim) -> (Tensor values , Tensor indices ) cumprod ( Tensor self , int dim , ScalarType? dtype=None ) → Tensor cumsum ( Tensor self , int dim=0 , ScalarType? dtype=None ) → Tensor data_ptr ( self : tensorplay._C.TensorBase ) → int deg2rad ( Tensor self ) → Tensor dense_dim ( self : tensorplay._C.TensorBase ) → int dequantize_per_channel ( Tensor self , Tensor scales , Tensor zero_points , int axis=0 ) → Tensor dequantize_per_tensor ( Tensor self , float scale , int zero_point ) → Tensor detach ( self : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase detach_ ( self : object ) → object diag ( Tensor self , int diagonal=0 ) → Tensor diag_embed ( Tensor self , int offset=0 , int dim1=0 , int dim2=1 ) → Tensor diagonal ( Tensor self , int offset=0 , int dim1=0 , int dim2=1 ) → Tensor diagonal_scatter ( Tensor self , Tensor src , int offset=0 , int dim1=0 , int dim2=1 ) → Tensor digamma ( Tensor self ) → Tensor dim ( self : tensorplay._C.TensorBase ) → int dist ( Tensor self , Tensor other , Scalar p=2 ) → Tensor div ( ) div_ ( ) div_ .Tensor(Tensor(a!) self, Tensor other) -> Tensor(a!) | div_ .Scalar(Tensor(a!) self, Scalar other) -> Tensor(a!) dot ( Tensor self , Tensor tensor ) → Tensor dsplit ( ) element_size ( self : tensorplay._C.TensorBase ) → int elu ( Tensor self , Scalar alpha=1 , Scalar scale=1 , Scalar input_scale=1 ) → Tensor eq ( ) equal ( Tensor self , Tensor other ) → bool erf ( Tensor self ) → Tensor erfc ( Tensor self ) → Tensor erfinv ( Tensor self ) → Tensor exp ( Tensor self ) → Tensor exp2 ( Tensor self ) → Tensor expand_as ( Tensor self , Tensor other ) → Tensor expm1 ( Tensor self ) → Tensor exponential_ ( Tensor(a!) self , float lambd=1.0) -> Tensor(a! ) fill_ ( ) fill_ .Scalar(Tensor(a!) self, Scalar value) -> Tensor(a!) fix ( Tensor self ) → Tensor flatten ( start_dim = 0 , end_dim = -1 ) Flattens a contiguous range of dims. floor ( Tensor self ) → Tensor frac ( Tensor self ) → Tensor gather ( Tensor self , int dim , Tensor index ) → Tensor gcd ( Tensor self , Tensor other ) → Tensor ge ( ) gelu ( Tensor self , str approximate="none" ) → Tensor geometric_ ( Tensor(a!) self , float p) -> Tensor(a! ) glu ( Tensor self , int dim=-1 ) → Tensor greater ( Tensor self , Tensor other ) → Tensor greater_equal ( Tensor self , Tensor other ) → Tensor gt ( ) hardshrink ( Tensor self , Scalar lambd=0.5 ) → Tensor hardsigmoid ( Tensor self ) → Tensor hardswish ( Tensor self ) → Tensor hardtanh ( Tensor self , Scalar min_val=-1 , Scalar max_val=1 ) → Tensor heaviside ( Tensor self , Tensor values ) → Tensor hsplit ( ) hypot ( Tensor self , Tensor other ) → Tensor i0e ( Tensor self ) → Tensor i1 ( Tensor self ) → Tensor i1e ( Tensor self ) → Tensor imag ( Tensor self ) → Tensor index_copy ( Tensor self , int dim , Tensor index , Tensor source ) → Tensor index_fill ( ) index_fill_ ( ) index_fill_ .Scalar(Tensor(a!) self, int dim, Tensor index, Scalar value) -> Tensor(a!) | index_fill_ .Tensor(Tensor(a!) self, int dim, Tensor index, Tensor value) -> Tensor(a!) index_put ( Tensor self , Tensor[] indices , Tensor values , bool accumulate=False ) → Tensor index_put_ ( Tensor(a!) self , Tensor[] indices , Tensor values , bool accumulate=False) -> Tensor(a! ) index_select ( Tensor self , int dim , Tensor index ) → Tensor inner ( Tensor self , Tensor other ) → Tensor is_channels_last ( self : tensorplay._C.TensorBase ) → bool is_channels_last_2d ( self : tensorplay._C.TensorBase ) → bool is_channels_last_3d ( self : tensorplay._C.TensorBase ) → bool is_coalesced ( self : tensorplay._C.TensorBase ) → bool is_complex ( self : tensorplay._C.TensorBase ) → bool is_contiguous ( * args , ** kwargs ) Overloaded function. is_contiguous(self: tensorplay._C.TensorBase) -> bool is_contiguous(self: tensorplay._C.TensorBase, memory_format: typing.SupportsInt | typing.SupportsIndex) -> bool is_floating_point ( self : tensorplay._C.TensorBase ) → bool is_pinned ( self : tensorplay._C.TensorBase ) → bool is_shared ( self : object ) → bool is_sparse_csr ( self : tensorplay._C.TensorBase ) → bool isclose ( Tensor self , Tensor other , float rtol=1e-05 , float atol=1e-08 , bool equal_nan=False ) → Tensor isfinite ( Tensor self ) → Tensor isinf ( Tensor self ) → Tensor isnan ( Tensor self ) → Tensor isneginf ( Tensor self ) → Tensor isposinf ( Tensor self ) → Tensor isreal ( Tensor self ) → Tensor itemsize ( self : tensorplay._C.TensorBase ) → int kthvalue ( Tensor self , int k , int dim=-1 , bool keepdim=False) -> (Tensor values , Tensor indices ) lcm ( Tensor self , Tensor other ) → Tensor le ( ) leaky_relu ( Tensor self , Scalar negative_slope=0.01 ) → Tensor lerp ( Tensor self , Tensor end , Scalar weight) -> Tensor | lerp.Tensor(Tensor self , Tensor end , Tensor weight ) → Tensor lerp_ ( ) lerp_ .Scalar(Tensor(a!) self, Tensor end, Scalar weight) -> Tensor(a!) | lerp_ .Tensor(Tensor(a!) self, Tensor end, Tensor weight) -> Tensor(a!) less ( Tensor self , Tensor other ) → Tensor less_equal ( Tensor self , Tensor other ) → Tensor lgamma ( Tensor self ) → Tensor log ( Tensor self ) → Tensor log10 ( Tensor self ) → Tensor log1p ( Tensor self ) → Tensor log2 ( Tensor self ) → Tensor log_normal_ ( Tensor(a!) self , float mean=1.0 , float std=2.0) -> Tensor(a! ) log_softmax ( Tensor self , int dim , ScalarType dtype=Undefined ) → Tensor logaddexp ( Tensor self , Tensor other ) → Tensor logaddexp2 ( Tensor self , Tensor other ) → Tensor logcumsumexp ( Tensor self , int dim , ScalarType? dtype=None ) → Tensor logical_not ( Tensor self ) → Tensor logical_or ( Tensor self , Tensor other ) → Tensor logical_xor ( Tensor self , Tensor other ) → Tensor logit ( Tensor self , Scalar? eps=None ) → Tensor logsumexp ( Tensor self , int dim , bool keepdim=False ) → Tensor lt ( ) masked_fill ( ) masked_fill_ ( ) masked_fill_ .Tensor(Tensor(a!) self, Tensor mask, Tensor value) -> Tensor(a!) | masked_fill_(Tensor(a!) self, Tensor mask, Scalar value) -> Tensor(a!) masked_scatter ( Tensor self , Tensor mask , Tensor source ) → Tensor masked_select ( Tensor self , Tensor mask ) → Tensor matmul ( Tensor self , Tensor other ) → Tensor max ( Tensor self) -> Tensor | max.dim(Tensor self , int[] dim , bool keepdim=false ) → Tensor mean ( Tensor self , * , ScalarType dtype=Undefined) -> Tensor | mean.dim(Tensor self , int[] dim , bool keepdim=false , * , ScalarType dtype=Undefined ) → Tensor median ( Tensor self ) → Tensor memory_format ( self : tensorplay._C.TensorBase ) → int min ( Tensor self) -> Tensor | min.dim(Tensor self , int[] dim , bool keepdim=false ) → Tensor mish ( Tensor self ) → Tensor mm ( Tensor self , Tensor mat2 ) → Tensor mode ( Tensor self , int dim=-1 , bool keepdim=False) -> (Tensor values , Tensor indices ) modified_bessel_i1 ( Tensor self ) → Tensor modified_bessel_k0 ( Tensor self ) → Tensor modified_bessel_k1 ( Tensor self ) → Tensor moveaxis ( ) movedim ( Tensor self , int[] source , int[] destination ) → Tensor msort ( Tensor self ) → Tensor mul ( ) mul_ ( ) mul_ .Tensor(Tensor(a!) self, Tensor other) -> Tensor(a!) | mul_ .Scalar(Tensor(a!) self, Scalar other) -> Tensor(a!) mv ( Tensor self , Tensor vec ) → Tensor nan_to_num ( Tensor self , Scalar nan=0.0 , Scalar? posinf=None , Scalar? neginf=None ) → Tensor nanmean ( Tensor self , int? dim=None , bool keepdim=False , * , ScalarType? dtype=None ) → Tensor nanmedian ( Tensor self ) → Tensor nansum ( Tensor self , int[] dim=[] , bool keepdim=False ) → Tensor narrow ( Tensor self , int dim , int start , int length ) → Tensor ndimension ( ) → int Alias for dim() ne ( ) neg ( Tensor self ) → Tensor neg_ ( Tensor(a!) self) -> Tensor(a! ) negative ( Tensor self ) → Tensor nextafter ( Tensor self , Tensor other ) → Tensor norm ( Tensor self , float p=2.0) -> Tensor | norm.dim(Tensor self , int[] dim , float p=2.0 , bool keepdim=false ) → Tensor normal_ ( Tensor(a!) self , float mean=0.0 , float std=1.0) -> Tensor(a! ) not_equal ( Tensor self , Tensor other ) → Tensor numel ( self : tensorplay._C.TensorBase ) → int numpy ( self : object ) → numpy.ndarray outer ( Tensor self , Tensor vec2 ) → Tensor pdist ( Tensor self , float p=2.0 ) → Tensor pin_memory ( self : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase poisson ( Tensor self ) → Tensor polygamma ( int n , Tensor self ) → Tensor positive ( Tensor self ) → Tensor pow ( ) prelu ( Tensor self , Tensor weight ) → Tensor prod ( Tensor self , * , ScalarType dtype=Undefined) -> Tensor | prod.dim_IntList(Tensor self , int[] dim , bool keepdim=false , * , ScalarType dtype=Undefined ) → Tensor quantize_per_channel ( Tensor self , Tensor scales , Tensor zero_points , int axis=0 ) → Tensor quantize_per_tensor ( Tensor self , float scale , int zero_point , int quant_min=-128 , int quant_max=127 ) → Tensor rad2deg ( Tensor self ) → Tensor random_ ( Tensor(a!) self , int low=0 , int high=0) -> Tensor(a! ) ravel ( Tensor self ) → Tensor reciprocal ( Tensor self ) → Tensor register_hook ( hook ) Registers a backward hook (torch’s Tensor.register_hook ). The hook is called every time a gradient with respect to this tensor is computed. It may modify the gradient by returning a replacement Tensor; returning None leaves the gradient unchanged. Hooks compose in registration order. Returns a RemovableHandle whose remove() method (or context-manager form) unregisters the hook. register_post_accumulate_grad_hook ( hook ) Registers a hook (torch’s Tensor.register_post_accumulate_grad_hook ). The hook runs after the gradient has been accumulated into self.grad . It receives the tensor (the parameter) and its return value is ignored; unlike register_hook() it cannot replace the gradient, but it may modify self.grad in place. Only leaf tensors that require grad and are used in the autograd graph support this hook. Returns a RemovableHandle . relu ( Tensor self ) → Tensor relu6 ( Tensor self ) → Tensor relu_ ( Tensor(a!) self) -> Tensor(a! ) renorm ( Tensor self , Scalar p , int dim , Scalar maxnorm ) → Tensor repeat ( Tensor self , int[] repeats ) → Tensor requires_grad_ ( self : object , requires_grad : bool = True ) → object reshape ( Tensor self , int[] shape ) → Tensor resize_ ( Tensor(a!) self , int[] size) -> Tensor(a! ) retain_grad ( self : tensorplay._C.TensorBase ) → None rot90 ( Tensor self , int k=1 , int[] dims=[] ) → Tensor round ( Tensor self ) → Tensor rsqrt ( Tensor self ) → Tensor rsqrt_ ( Tensor(a!) self) -> Tensor(a! ) scaled_modified_bessel_k0 ( Tensor self ) → Tensor scaled_modified_bessel_k1 ( Tensor self ) → Tensor scatter ( ) scatter_ ( ) scatter_ .src(Tensor(a!) self, int dim, Tensor index, Tensor src) -> Tensor(a!) | scatter_ .value(Tensor(a!) self, int dim, Tensor index, Scalar value) -> Tensor(a!) scatter_add ( Tensor self , int dim , Tensor index , Tensor src ) → Tensor scatter_add_ ( Tensor(a!) self , int dim , Tensor index , Tensor src) -> Tensor(a! ) select ( Tensor self , int dim , int index ) → Tensor select_scatter ( Tensor self , Tensor src , int dim , int index ) → Tensor selu ( Tensor self ) → Tensor sgn ( Tensor self ) → Tensor share_memory_ ( self : object ) → object sigmoid ( Tensor self ) → Tensor sign ( Tensor self ) → Tensor signbit ( Tensor self ) → Tensor silu ( Tensor self ) → Tensor sin ( Tensor self ) → Tensor sinc ( Tensor self ) → Tensor sinh ( Tensor self ) → Tensor size ( * args , ** kwargs ) Overloaded function. size(self: tensorplay._C.TensorBase) -> tensorplay._C.Size size(self: tensorplay._C.TensorBase, arg0: typing.SupportsInt | typing.SupportsIndex) -> int slice ( Tensor self , int dim=0 , int? start=None , int? end=None , int step=1 ) → Tensor slice_scatter ( Tensor self , Tensor src , int dim=0 , int? start=None , int? end=None , int step=1 ) → Tensor softmax ( Tensor self , int dim , ScalarType dtype=Undefined ) → Tensor softplus ( Tensor self , Scalar beta=1 , Scalar threshold=20 ) → Tensor softshrink ( Tensor self , Scalar lambd=0.5 ) → Tensor sort ( Tensor self , int dim=-1 , bool descending=False) -> (Tensor values , Tensor indices ) sparse_dim ( self : tensorplay._C.TensorBase ) → int sparse_mask ( self : tensorplay._C.TensorBase , mask : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase sparse_sum ( Tensor self , int[]? dim=None , ScalarType? dtype=None ) → Tensor spherical_bessel_j0 ( Tensor self ) → Tensor split ( Tensor self , int split_size , int dim=0) -> Tensor[] | split.sizes(Tensor self , int[] split_sizes , int dim=0 ) → Tensor[] split_with_sizes ( Tensor self , int[] split_sizes , int dim=0 ) → Tensor[] sqrt ( Tensor self ) → Tensor sqrt_ ( Tensor(a!) self) -> Tensor(a! ) square ( Tensor self ) → Tensor squeeze ( Tensor self) -> Tensor | squeeze.dim(Tensor self , int dim ) → Tensor std ( Tensor self , int correction=1) -> Tensor | std.dim(Tensor self , int[] dim , int correction=1 , bool keepdim=false ) → Tensor stride ( * args , ** kwargs ) Overloaded function. stride(self: tensorplay._C.TensorBase) -> tuple stride(self: tensorplay._C.TensorBase, arg0: typing.SupportsInt | typing.SupportsIndex) -> int sub ( ) sub_ ( ) sub_ .Tensor(Tensor(a!) self, Tensor other, * , Scalar alpha=1) -> Tensor(a!) | sub_ .Scalar(Tensor(a!) self, Scalar other, Scalar alpha=1) -> Tensor(a!) sum ( Tensor self , * , ScalarType dtype=Undefined) -> Tensor | sum.dim_IntList(Tensor self , int[] dim , bool keepdim=false , * , ScalarType dtype=Undefined ) → Tensor svd ( Tensor self , bool some=True , bool compute_uv=True) -> (Tensor U , Tensor S , Tensor V ) swapaxes ( Tensor self , int axis0 , int axis1 ) → Tensor swapdims ( Tensor self , int dim0 , int dim1 ) → Tensor t ( ) Returns the transpose of the tensor. Aliased to transpose(0, 1) to ensure correct autograd behavior (TransposeBackward). take ( Tensor self , Tensor index ) → Tensor take_along_dim ( Tensor self , Tensor indices , int? dim=None ) → Tensor tan ( Tensor self ) → Tensor tanh ( Tensor self ) → Tensor tensor_split ( ) tile ( Tensor self , int[] dims ) → Tensor to ( * args , ** kwargs ) Overloaded function. to(self: tensorplay._C.TensorBase, dtype: tensorplay._C.DType, non_blocking: bool = False, copy: bool = False) -> tensorplay._C.TensorBase to(self: tensorplay._C.TensorBase, device: tensorplay._C.Device, non_blocking: bool = False, copy: bool = False) -> tensorplay._C.TensorBase to(self: tensorplay._C.TensorBase, device: tensorplay._C.Device, dtype: tensorplay._C.DType, non_blocking: bool = False, copy: bool = False) -> tensorplay._C.TensorBase to_dense ( Tensor self ) → Tensor to_sparse ( Tensor self ) → Tensor to_sparse_csr ( Tensor self ) → Tensor tolist ( self : tensorplay._C.TensorBase ) → object trace ( Tensor self ) → Tensor transpose ( Tensor self , int dim0 , int dim1 ) → Tensor triangular_solve ( Tensor self , Tensor A , bool upper=False , bool transpose=False , bool unitriangular=False) -> (Tensor solution , Tensor cloned_coefficient ) tril ( Tensor self , int diagonal=0 ) → Tensor triu ( Tensor self , int diagonal=0 ) → Tensor trunc ( Tensor self ) → Tensor type ( dtype = None , non_blocking = False , ** kwargs ) Returns the type if dtype is not provided, else casts this object to the specified type. unbind ( Tensor self , int dim=0 ) → Tensor[] unflatten ( dim , sizes ) Expands a dimension of the input tensor over multiple dimensions. unfold ( dimension , size , step ) Returns a view of the original tensor which contains all slices of size size from self in the dimension dimension , stepping by step (torch’s Tensor.unfold ). Port of aten/src/ATen/native/TensorShape.cpp : the view appends a new trailing dimension of length size and re-strides dimension by step . uniform_ ( Tensor(a!) self , float from=0.0 , float to=1.0) -> Tensor(a! ) unsqueeze ( Tensor self , int dim ) → Tensor values ( self : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase var ( Tensor self , int correction=1) -> Tensor | var.dim(Tensor self , int[] dim , int correction=1 , bool keepdim=false ) → Tensor vdot ( Tensor self , Tensor other ) → Tensor view ( Tensor self , int[] shape ) → Tensor vsplit ( ) zero_ ( Tensor(a!) self) -> Tensor(a! ) zeta ( Tensor self , Tensor other ) → Tensor # tensorplay.nn.parameter.is_lazy URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.parameter.is_lazy.html tensorplay.nn.parameter.is_lazy tensorplay.nn.parameter. is_lazy ( param : Any ) → bool [source] Returns whether param is an UninitializedParameter or UninitializedBuffer . Parameters : param ( Any ) – the input to check. # RNNCellBase URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.RNNCellBase.html RNNCellBase class tensorplay.nn. RNNCellBase ( input_size : int , hidden_size : int , bias : bool , num_chunks : int , device = None , dtype = None ) [source] add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( * input : Any ) → None Define the computation performed at every call. Should be overridden by all subclasses. Note Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # UninitializedBuffer URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.UninitializedBuffer.html UninitializedBuffer class tensorplay.nn. UninitializedBuffer ( requires_grad = False , device = None , dtype = None , persistent = True ) [source] A buffer that is not initialized. Uninitialized Buffer is a a special case of tensorplay.Tensor where the shape of the data is still unknown. Unlike a tensorplay.Tensor , uninitialized parameters hold no data and attempting to access some properties, like their shape, will throw a runtime error. The only operations that can be performed on an uninitialized parameter are changing its datatype, moving it to a different device and converting it to a regular tensorplay.Tensor . The default device or dtype to use when the buffer is materialized can be set during construction using e.g. device='cuda' . abs ( Tensor self ) → Tensor abs_ ( Tensor(a!) self) -> Tensor(a! ) acos ( Tensor self ) → Tensor acosh ( Tensor self ) → Tensor add ( ) add_ ( ) add_ .Tensor(Tensor(a!) self, Tensor other, * , Scalar alpha=1) -> Tensor(a!) | add_ .Scalar(Tensor(a!) self, Scalar other, Scalar alpha=1) -> Tensor(a!) addbmm ( Tensor self , Tensor batch1 , Tensor batch2 , * , Scalar beta=1 , Scalar alpha=1 ) → Tensor addcdiv ( Tensor self , Tensor tensor1 , Tensor tensor2 , * , Scalar value=1 ) → Tensor addcdiv_ ( Tensor(a!) self , Tensor tensor1 , Tensor tensor2 , * , Scalar value=1) -> Tensor(a! ) addcmul ( Tensor self , Tensor tensor1 , Tensor tensor2 , * , Scalar value=1 ) → Tensor addcmul_ ( Tensor(a!) self , Tensor tensor1 , Tensor tensor2 , * , Scalar value=1) -> Tensor(a! ) addmv ( Tensor self , Tensor mat , Tensor vec , * , Scalar beta=1 , Scalar alpha=1 ) → Tensor addr ( Tensor self , Tensor vec1 , Tensor vec2 , * , Scalar beta=1 , Scalar alpha=1 ) → Tensor airy_ai ( Tensor self ) → Tensor all ( Tensor self) -> Tensor | all.dim(Tensor self , int[] dim , bool keepdim=false ) → Tensor allclose ( Tensor self , Tensor other , float rtol=1e-05 , float atol=1e-08 , bool equal_nan=False ) → bool angle ( Tensor self ) → Tensor any ( Tensor self) -> Tensor | any.dim(Tensor self , int[] dim , bool keepdim=false ) → Tensor argmax ( Tensor self , int? dim=None , bool keepdim=False ) → Tensor argmin ( Tensor self , int? dim=None , bool keepdim=False ) → Tensor argsort ( Tensor self , int dim=-1 , bool descending=False ) → Tensor argwhere ( Tensor self ) → Tensor as_strided ( self : tensorplay._C.TensorBase , size : collections.abc.Sequence [ SupportsInt | SupportsIndex ] , stride : collections.abc.Sequence [ SupportsInt | SupportsIndex ] , storage_offset : SupportsInt | SupportsIndex | None = None ) → tensorplay._C.TensorBase asin ( Tensor self ) → Tensor asinh ( Tensor self ) → Tensor atan ( Tensor self ) → Tensor atan2 ( Tensor self , Tensor other ) → Tensor atanh ( Tensor self ) → Tensor backward ( self : tensorplay._C.TensorBase , gradient : tensorplay._C.TensorBase | None = None , retain_graph : bool | None = None , create_graph : bool = False ) → None bernoulli ( Tensor self ) → Tensor bernoulli_ ( Tensor(a!) self) -> Tensor(a! ) bessel_j0 ( Tensor self ) → Tensor bessel_j1 ( Tensor self ) → Tensor bessel_y0 ( Tensor self ) → Tensor bessel_y1 ( Tensor self ) → Tensor bincount ( Tensor self , Tensor? weights=None , int minlength=0 ) → Tensor bitwise_and ( ) bitwise_left_shift ( ) bitwise_not ( Tensor self ) → Tensor bitwise_or ( ) bitwise_right_shift ( ) bitwise_xor ( ) bmm ( Tensor self , Tensor mat2 ) → Tensor broadcast_to ( Tensor self , int[] size ) → Tensor cauchy_ ( Tensor(a!) self , float median=0.0 , float sigma=1.0) -> Tensor(a! ) ceil ( Tensor self ) → Tensor celu ( Tensor self , Scalar alpha=1.0 ) → Tensor channel_shuffle ( Tensor self , int groups ) → Tensor cholesky ( Tensor self , bool upper=False ) → Tensor cholesky_inverse ( Tensor self , bool upper=False ) → Tensor cholesky_solve ( Tensor self , Tensor input2 , bool upper=False ) → Tensor chunk ( Tensor self , int chunks , int dim=0 ) → Tensor[] clamp ( Tensor self , Scalar? min=None , Scalar? max=None ) → Tensor clamp_ ( Tensor(a!) self , Scalar? min=None , Scalar? max=None) -> Tensor(a! ) clip ( Tensor self , Scalar? min=None , Scalar? max=None ) → Tensor clone ( Tensor self , * , MemoryFormat? memory_format=None ) → Tensor coalesce ( self : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase col_indices ( self : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase conj ( Tensor self ) → Tensor contiguous ( Tensor self , * , MemoryFormat memory_format=Contiguous ) → Tensor copy_ ( Tensor(a!) self , Tensor src , bool non_blocking=False) -> Tensor(a! ) cos ( Tensor self ) → Tensor cosh ( Tensor self ) → Tensor cpu ( ) Returns a copy of this object in CPU memory. If this object is already in CPU memory, then no copy is performed and the original object is returned. crow_indices ( self : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase cuda ( device = None , non_blocking = False ) Returns a copy of this object in CUDA memory. If this object is already in CUDA memory and on the correct device, then no copy is performed and the original object is returned. cummax ( Tensor self , int dim) -> (Tensor values , Tensor indices ) cummin ( Tensor self , int dim) -> (Tensor values , Tensor indices ) cumprod ( Tensor self , int dim , ScalarType? dtype=None ) → Tensor cumsum ( Tensor self , int dim=0 , ScalarType? dtype=None ) → Tensor data_ptr ( self : tensorplay._C.TensorBase ) → int deg2rad ( Tensor self ) → Tensor dense_dim ( self : tensorplay._C.TensorBase ) → int dequantize_per_channel ( Tensor self , Tensor scales , Tensor zero_points , int axis=0 ) → Tensor dequantize_per_tensor ( Tensor self , float scale , int zero_point ) → Tensor detach ( self : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase detach_ ( self : object ) → object diag ( Tensor self , int diagonal=0 ) → Tensor diag_embed ( Tensor self , int offset=0 , int dim1=0 , int dim2=1 ) → Tensor diagonal ( Tensor self , int offset=0 , int dim1=0 , int dim2=1 ) → Tensor diagonal_scatter ( Tensor self , Tensor src , int offset=0 , int dim1=0 , int dim2=1 ) → Tensor digamma ( Tensor self ) → Tensor dim ( self : tensorplay._C.TensorBase ) → int dist ( Tensor self , Tensor other , Scalar p=2 ) → Tensor div ( ) div_ ( ) div_ .Tensor(Tensor(a!) self, Tensor other) -> Tensor(a!) | div_ .Scalar(Tensor(a!) self, Scalar other) -> Tensor(a!) dot ( Tensor self , Tensor tensor ) → Tensor dsplit ( ) element_size ( self : tensorplay._C.TensorBase ) → int elu ( Tensor self , Scalar alpha=1 , Scalar scale=1 , Scalar input_scale=1 ) → Tensor eq ( ) equal ( Tensor self , Tensor other ) → bool erf ( Tensor self ) → Tensor erfc ( Tensor self ) → Tensor erfinv ( Tensor self ) → Tensor exp ( Tensor self ) → Tensor exp2 ( Tensor self ) → Tensor expand_as ( Tensor self , Tensor other ) → Tensor expm1 ( Tensor self ) → Tensor exponential_ ( Tensor(a!) self , float lambd=1.0) -> Tensor(a! ) fill_ ( ) fill_ .Scalar(Tensor(a!) self, Scalar value) -> Tensor(a!) fix ( Tensor self ) → Tensor flatten ( start_dim = 0 , end_dim = -1 ) Flattens a contiguous range of dims. floor ( Tensor self ) → Tensor frac ( Tensor self ) → Tensor gather ( Tensor self , int dim , Tensor index ) → Tensor gcd ( Tensor self , Tensor other ) → Tensor ge ( ) gelu ( Tensor self , str approximate="none" ) → Tensor geometric_ ( Tensor(a!) self , float p) -> Tensor(a! ) glu ( Tensor self , int dim=-1 ) → Tensor greater ( Tensor self , Tensor other ) → Tensor greater_equal ( Tensor self , Tensor other ) → Tensor gt ( ) hardshrink ( Tensor self , Scalar lambd=0.5 ) → Tensor hardsigmoid ( Tensor self ) → Tensor hardswish ( Tensor self ) → Tensor hardtanh ( Tensor self , Scalar min_val=-1 , Scalar max_val=1 ) → Tensor heaviside ( Tensor self , Tensor values ) → Tensor hsplit ( ) hypot ( Tensor self , Tensor other ) → Tensor i0e ( Tensor self ) → Tensor i1 ( Tensor self ) → Tensor i1e ( Tensor self ) → Tensor imag ( Tensor self ) → Tensor index_copy ( Tensor self , int dim , Tensor index , Tensor source ) → Tensor index_fill ( ) index_fill_ ( ) index_fill_ .Scalar(Tensor(a!) self, int dim, Tensor index, Scalar value) -> Tensor(a!) | index_fill_ .Tensor(Tensor(a!) self, int dim, Tensor index, Tensor value) -> Tensor(a!) index_put ( Tensor self , Tensor[] indices , Tensor values , bool accumulate=False ) → Tensor index_put_ ( Tensor(a!) self , Tensor[] indices , Tensor values , bool accumulate=False) -> Tensor(a! ) index_select ( Tensor self , int dim , Tensor index ) → Tensor inner ( Tensor self , Tensor other ) → Tensor is_channels_last ( self : tensorplay._C.TensorBase ) → bool is_channels_last_2d ( self : tensorplay._C.TensorBase ) → bool is_channels_last_3d ( self : tensorplay._C.TensorBase ) → bool is_coalesced ( self : tensorplay._C.TensorBase ) → bool is_complex ( self : tensorplay._C.TensorBase ) → bool is_contiguous ( * args , ** kwargs ) Overloaded function. is_contiguous(self: tensorplay._C.TensorBase) -> bool is_contiguous(self: tensorplay._C.TensorBase, memory_format: typing.SupportsInt | typing.SupportsIndex) -> bool is_floating_point ( self : tensorplay._C.TensorBase ) → bool is_pinned ( self : tensorplay._C.TensorBase ) → bool is_shared ( self : object ) → bool is_sparse_csr ( self : tensorplay._C.TensorBase ) → bool isclose ( Tensor self , Tensor other , float rtol=1e-05 , float atol=1e-08 , bool equal_nan=False ) → Tensor isfinite ( Tensor self ) → Tensor isinf ( Tensor self ) → Tensor isnan ( Tensor self ) → Tensor isneginf ( Tensor self ) → Tensor isposinf ( Tensor self ) → Tensor isreal ( Tensor self ) → Tensor itemsize ( self : tensorplay._C.TensorBase ) → int kthvalue ( Tensor self , int k , int dim=-1 , bool keepdim=False) -> (Tensor values , Tensor indices ) lcm ( Tensor self , Tensor other ) → Tensor le ( ) leaky_relu ( Tensor self , Scalar negative_slope=0.01 ) → Tensor lerp ( Tensor self , Tensor end , Scalar weight) -> Tensor | lerp.Tensor(Tensor self , Tensor end , Tensor weight ) → Tensor lerp_ ( ) lerp_ .Scalar(Tensor(a!) self, Tensor end, Scalar weight) -> Tensor(a!) | lerp_ .Tensor(Tensor(a!) self, Tensor end, Tensor weight) -> Tensor(a!) less ( Tensor self , Tensor other ) → Tensor less_equal ( Tensor self , Tensor other ) → Tensor lgamma ( Tensor self ) → Tensor log ( Tensor self ) → Tensor log10 ( Tensor self ) → Tensor log1p ( Tensor self ) → Tensor log2 ( Tensor self ) → Tensor log_normal_ ( Tensor(a!) self , float mean=1.0 , float std=2.0) -> Tensor(a! ) log_softmax ( Tensor self , int dim , ScalarType dtype=Undefined ) → Tensor logaddexp ( Tensor self , Tensor other ) → Tensor logaddexp2 ( Tensor self , Tensor other ) → Tensor logcumsumexp ( Tensor self , int dim , ScalarType? dtype=None ) → Tensor logical_not ( Tensor self ) → Tensor logical_or ( Tensor self , Tensor other ) → Tensor logical_xor ( Tensor self , Tensor other ) → Tensor logit ( Tensor self , Scalar? eps=None ) → Tensor logsumexp ( Tensor self , int dim , bool keepdim=False ) → Tensor lt ( ) masked_fill ( ) masked_fill_ ( ) masked_fill_ .Tensor(Tensor(a!) self, Tensor mask, Tensor value) -> Tensor(a!) | masked_fill_(Tensor(a!) self, Tensor mask, Scalar value) -> Tensor(a!) masked_scatter ( Tensor self , Tensor mask , Tensor source ) → Tensor masked_select ( Tensor self , Tensor mask ) → Tensor materialize ( shape , device = None , dtype = None ) Create a Parameter or Tensor with the same properties of the uninitialized one. Given a shape, it materializes a parameter in the same device and with the same dtype as the current one or the specified ones in the arguments. Parameters : shape – (tuple): the shape for the materialized tensor. device ( tensorplay.device ) – the desired device of the parameters and buffers in this module. Optional. dtype ( tensorplay.dtype ) – the desired floating point type of the floating point parameters and buffers in this module. Optional. matmul ( Tensor self , Tensor other ) → Tensor max ( Tensor self) -> Tensor | max.dim(Tensor self , int[] dim , bool keepdim=false ) → Tensor mean ( Tensor self , * , ScalarType dtype=Undefined) -> Tensor | mean.dim(Tensor self , int[] dim , bool keepdim=false , * , ScalarType dtype=Undefined ) → Tensor median ( Tensor self ) → Tensor memory_format ( self : tensorplay._C.TensorBase ) → int min ( Tensor self) -> Tensor | min.dim(Tensor self , int[] dim , bool keepdim=false ) → Tensor mish ( Tensor self ) → Tensor mm ( Tensor self , Tensor mat2 ) → Tensor mode ( Tensor self , int dim=-1 , bool keepdim=False) -> (Tensor values , Tensor indices ) modified_bessel_i1 ( Tensor self ) → Tensor modified_bessel_k0 ( Tensor self ) → Tensor modified_bessel_k1 ( Tensor self ) → Tensor moveaxis ( ) movedim ( Tensor self , int[] source , int[] destination ) → Tensor msort ( Tensor self ) → Tensor mul ( ) mul_ ( ) mul_ .Tensor(Tensor(a!) self, Tensor other) -> Tensor(a!) | mul_ .Scalar(Tensor(a!) self, Scalar other) -> Tensor(a!) mv ( Tensor self , Tensor vec ) → Tensor nan_to_num ( Tensor self , Scalar nan=0.0 , Scalar? posinf=None , Scalar? neginf=None ) → Tensor nanmean ( Tensor self , int? dim=None , bool keepdim=False , * , ScalarType? dtype=None ) → Tensor nanmedian ( Tensor self ) → Tensor nansum ( Tensor self , int[] dim=[] , bool keepdim=False ) → Tensor narrow ( Tensor self , int dim , int start , int length ) → Tensor ndimension ( ) → int Alias for dim() ne ( ) neg ( Tensor self ) → Tensor neg_ ( Tensor(a!) self) -> Tensor(a! ) negative ( Tensor self ) → Tensor nextafter ( Tensor self , Tensor other ) → Tensor norm ( Tensor self , float p=2.0) -> Tensor | norm.dim(Tensor self , int[] dim , float p=2.0 , bool keepdim=false ) → Tensor normal_ ( Tensor(a!) self , float mean=0.0 , float std=1.0) -> Tensor(a! ) not_equal ( Tensor self , Tensor other ) → Tensor numel ( self : tensorplay._C.TensorBase ) → int numpy ( self : object ) → numpy.ndarray outer ( Tensor self , Tensor vec2 ) → Tensor pdist ( Tensor self , float p=2.0 ) → Tensor pin_memory ( self : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase poisson ( Tensor self ) → Tensor polygamma ( int n , Tensor self ) → Tensor positive ( Tensor self ) → Tensor pow ( ) prelu ( Tensor self , Tensor weight ) → Tensor prod ( Tensor self , * , ScalarType dtype=Undefined) -> Tensor | prod.dim_IntList(Tensor self , int[] dim , bool keepdim=false , * , ScalarType dtype=Undefined ) → Tensor quantize_per_channel ( Tensor self , Tensor scales , Tensor zero_points , int axis=0 ) → Tensor quantize_per_tensor ( Tensor self , float scale , int zero_point , int quant_min=-128 , int quant_max=127 ) → Tensor rad2deg ( Tensor self ) → Tensor random_ ( Tensor(a!) self , int low=0 , int high=0) -> Tensor(a! ) ravel ( Tensor self ) → Tensor reciprocal ( Tensor self ) → Tensor register_hook ( hook ) Registers a backward hook (torch’s Tensor.register_hook ). The hook is called every time a gradient with respect to this tensor is computed. It may modify the gradient by returning a replacement Tensor; returning None leaves the gradient unchanged. Hooks compose in registration order. Returns a RemovableHandle whose remove() method (or context-manager form) unregisters the hook. register_post_accumulate_grad_hook ( hook ) Registers a hook (torch’s Tensor.register_post_accumulate_grad_hook ). The hook runs after the gradient has been accumulated into self.grad . It receives the tensor (the parameter) and its return value is ignored; unlike register_hook() it cannot replace the gradient, but it may modify self.grad in place. Only leaf tensors that require grad and are used in the autograd graph support this hook. Returns a RemovableHandle . relu ( Tensor self ) → Tensor relu6 ( Tensor self ) → Tensor relu_ ( Tensor(a!) self) -> Tensor(a! ) renorm ( Tensor self , Scalar p , int dim , Scalar maxnorm ) → Tensor repeat ( Tensor self , int[] repeats ) → Tensor requires_grad_ ( self : object , requires_grad : bool = True ) → object reshape ( Tensor self , int[] shape ) → Tensor resize_ ( Tensor(a!) self , int[] size) -> Tensor(a! ) retain_grad ( self : tensorplay._C.TensorBase ) → None rot90 ( Tensor self , int k=1 , int[] dims=[] ) → Tensor round ( Tensor self ) → Tensor rsqrt ( Tensor self ) → Tensor rsqrt_ ( Tensor(a!) self) -> Tensor(a! ) scaled_modified_bessel_k0 ( Tensor self ) → Tensor scaled_modified_bessel_k1 ( Tensor self ) → Tensor scatter ( ) scatter_ ( ) scatter_ .src(Tensor(a!) self, int dim, Tensor index, Tensor src) -> Tensor(a!) | scatter_ .value(Tensor(a!) self, int dim, Tensor index, Scalar value) -> Tensor(a!) scatter_add ( Tensor self , int dim , Tensor index , Tensor src ) → Tensor scatter_add_ ( Tensor(a!) self , int dim , Tensor index , Tensor src) -> Tensor(a! ) select ( Tensor self , int dim , int index ) → Tensor select_scatter ( Tensor self , Tensor src , int dim , int index ) → Tensor selu ( Tensor self ) → Tensor sgn ( Tensor self ) → Tensor sigmoid ( Tensor self ) → Tensor sign ( Tensor self ) → Tensor signbit ( Tensor self ) → Tensor silu ( Tensor self ) → Tensor sin ( Tensor self ) → Tensor sinc ( Tensor self ) → Tensor sinh ( Tensor self ) → Tensor size ( * args , ** kwargs ) Overloaded function. size(self: tensorplay._C.TensorBase) -> tensorplay._C.Size size(self: tensorplay._C.TensorBase, arg0: typing.SupportsInt | typing.SupportsIndex) -> int slice ( Tensor self , int dim=0 , int? start=None , int? end=None , int step=1 ) → Tensor slice_scatter ( Tensor self , Tensor src , int dim=0 , int? start=None , int? end=None , int step=1 ) → Tensor softmax ( Tensor self , int dim , ScalarType dtype=Undefined ) → Tensor softplus ( Tensor self , Scalar beta=1 , Scalar threshold=20 ) → Tensor softshrink ( Tensor self , Scalar lambd=0.5 ) → Tensor sort ( Tensor self , int dim=-1 , bool descending=False) -> (Tensor values , Tensor indices ) sparse_dim ( self : tensorplay._C.TensorBase ) → int sparse_mask ( self : tensorplay._C.TensorBase , mask : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase sparse_sum ( Tensor self , int[]? dim=None , ScalarType? dtype=None ) → Tensor spherical_bessel_j0 ( Tensor self ) → Tensor split ( Tensor self , int split_size , int dim=0) -> Tensor[] | split.sizes(Tensor self , int[] split_sizes , int dim=0 ) → Tensor[] split_with_sizes ( Tensor self , int[] split_sizes , int dim=0 ) → Tensor[] sqrt ( Tensor self ) → Tensor sqrt_ ( Tensor(a!) self) -> Tensor(a! ) square ( Tensor self ) → Tensor squeeze ( Tensor self) -> Tensor | squeeze.dim(Tensor self , int dim ) → Tensor std ( Tensor self , int correction=1) -> Tensor | std.dim(Tensor self , int[] dim , int correction=1 , bool keepdim=false ) → Tensor stride ( * args , ** kwargs ) Overloaded function. stride(self: tensorplay._C.TensorBase) -> tuple stride(self: tensorplay._C.TensorBase, arg0: typing.SupportsInt | typing.SupportsIndex) -> int sub ( ) sub_ ( ) sub_ .Tensor(Tensor(a!) self, Tensor other, * , Scalar alpha=1) -> Tensor(a!) | sub_ .Scalar(Tensor(a!) self, Scalar other, Scalar alpha=1) -> Tensor(a!) sum ( Tensor self , * , ScalarType dtype=Undefined) -> Tensor | sum.dim_IntList(Tensor self , int[] dim , bool keepdim=false , * , ScalarType dtype=Undefined ) → Tensor svd ( Tensor self , bool some=True , bool compute_uv=True) -> (Tensor U , Tensor S , Tensor V ) swapaxes ( Tensor self , int axis0 , int axis1 ) → Tensor swapdims ( Tensor self , int dim0 , int dim1 ) → Tensor t ( ) Returns the transpose of the tensor. Aliased to transpose(0, 1) to ensure correct autograd behavior (TransposeBackward). take ( Tensor self , Tensor index ) → Tensor take_along_dim ( Tensor self , Tensor indices , int? dim=None ) → Tensor tan ( Tensor self ) → Tensor tanh ( Tensor self ) → Tensor tensor_split ( ) tile ( Tensor self , int[] dims ) → Tensor to ( * args , ** kwargs ) Overloaded function. to(self: tensorplay._C.TensorBase, dtype: tensorplay._C.DType, non_blocking: bool = False, copy: bool = False) -> tensorplay._C.TensorBase to(self: tensorplay._C.TensorBase, device: tensorplay._C.Device, non_blocking: bool = False, copy: bool = False) -> tensorplay._C.TensorBase to(self: tensorplay._C.TensorBase, device: tensorplay._C.Device, dtype: tensorplay._C.DType, non_blocking: bool = False, copy: bool = False) -> tensorplay._C.TensorBase to_dense ( Tensor self ) → Tensor to_sparse ( Tensor self ) → Tensor to_sparse_csr ( Tensor self ) → Tensor tolist ( self : tensorplay._C.TensorBase ) → object trace ( Tensor self ) → Tensor transpose ( Tensor self , int dim0 , int dim1 ) → Tensor triangular_solve ( Tensor self , Tensor A , bool upper=False , bool transpose=False , bool unitriangular=False) -> (Tensor solution , Tensor cloned_coefficient ) tril ( Tensor self , int diagonal=0 ) → Tensor triu ( Tensor self , int diagonal=0 ) → Tensor trunc ( Tensor self ) → Tensor type ( dtype = None , non_blocking = False , ** kwargs ) Returns the type if dtype is not provided, else casts this object to the specified type. unbind ( Tensor self , int dim=0 ) → Tensor[] unflatten ( dim , sizes ) Expands a dimension of the input tensor over multiple dimensions. unfold ( dimension , size , step ) Returns a view of the original tensor which contains all slices of size size from self in the dimension dimension , stepping by step (torch’s Tensor.unfold ). Port of aten/src/ATen/native/TensorShape.cpp : the view appends a new trailing dimension of length size and re-strides dimension by step . uniform_ ( Tensor(a!) self , float from=0.0 , float to=1.0) -> Tensor(a! ) unsqueeze ( Tensor self , int dim ) → Tensor values ( self : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase var ( Tensor self , int correction=1) -> Tensor | var.dim(Tensor self , int[] dim , int correction=1 , bool keepdim=false ) → Tensor vdot ( Tensor self , Tensor other ) → Tensor view ( Tensor self , int[] shape ) → Tensor vsplit ( ) zero_ ( Tensor(a!) self) -> Tensor(a! ) zeta ( Tensor self , Tensor other ) → Tensor # UninitializedParameter URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.UninitializedParameter.html UninitializedParameter class tensorplay.nn. UninitializedParameter ( requires_grad = True , device = None , dtype = None ) [source] A parameter that is not initialized. Uninitialized Parameters are a special case of tensorplay.nn.Parameter where the shape of the data is still unknown. Unlike a tensorplay.nn.Parameter , uninitialized parameters hold no data and attempting to access some properties, like their shape, will throw a runtime error. The only operations that can be performed on a uninitialized parameter are changing its datatype, moving it to a different device and converting it to a regular tensorplay.nn.Parameter . The default device or dtype to use when the parameter is materialized can be set during construction using e.g. device='cuda' . abs ( Tensor self ) → Tensor abs_ ( Tensor(a!) self) -> Tensor(a! ) acos ( Tensor self ) → Tensor acosh ( Tensor self ) → Tensor add ( ) add_ ( ) add_ .Tensor(Tensor(a!) self, Tensor other, * , Scalar alpha=1) -> Tensor(a!) | add_ .Scalar(Tensor(a!) self, Scalar other, Scalar alpha=1) -> Tensor(a!) addbmm ( Tensor self , Tensor batch1 , Tensor batch2 , * , Scalar beta=1 , Scalar alpha=1 ) → Tensor addcdiv ( Tensor self , Tensor tensor1 , Tensor tensor2 , * , Scalar value=1 ) → Tensor addcdiv_ ( Tensor(a!) self , Tensor tensor1 , Tensor tensor2 , * , Scalar value=1) -> Tensor(a! ) addcmul ( Tensor self , Tensor tensor1 , Tensor tensor2 , * , Scalar value=1 ) → Tensor addcmul_ ( Tensor(a!) self , Tensor tensor1 , Tensor tensor2 , * , Scalar value=1) -> Tensor(a! ) addmv ( Tensor self , Tensor mat , Tensor vec , * , Scalar beta=1 , Scalar alpha=1 ) → Tensor addr ( Tensor self , Tensor vec1 , Tensor vec2 , * , Scalar beta=1 , Scalar alpha=1 ) → Tensor airy_ai ( Tensor self ) → Tensor all ( Tensor self) -> Tensor | all.dim(Tensor self , int[] dim , bool keepdim=false ) → Tensor allclose ( Tensor self , Tensor other , float rtol=1e-05 , float atol=1e-08 , bool equal_nan=False ) → bool angle ( Tensor self ) → Tensor any ( Tensor self) -> Tensor | any.dim(Tensor self , int[] dim , bool keepdim=false ) → Tensor argmax ( Tensor self , int? dim=None , bool keepdim=False ) → Tensor argmin ( Tensor self , int? dim=None , bool keepdim=False ) → Tensor argsort ( Tensor self , int dim=-1 , bool descending=False ) → Tensor argwhere ( Tensor self ) → Tensor as_strided ( self : tensorplay._C.TensorBase , size : collections.abc.Sequence [ SupportsInt | SupportsIndex ] , stride : collections.abc.Sequence [ SupportsInt | SupportsIndex ] , storage_offset : SupportsInt | SupportsIndex | None = None ) → tensorplay._C.TensorBase asin ( Tensor self ) → Tensor asinh ( Tensor self ) → Tensor atan ( Tensor self ) → Tensor atan2 ( Tensor self , Tensor other ) → Tensor atanh ( Tensor self ) → Tensor backward ( self : tensorplay._C.TensorBase , gradient : tensorplay._C.TensorBase | None = None , retain_graph : bool | None = None , create_graph : bool = False ) → None bernoulli ( Tensor self ) → Tensor bernoulli_ ( Tensor(a!) self) -> Tensor(a! ) bessel_j0 ( Tensor self ) → Tensor bessel_j1 ( Tensor self ) → Tensor bessel_y0 ( Tensor self ) → Tensor bessel_y1 ( Tensor self ) → Tensor bincount ( Tensor self , Tensor? weights=None , int minlength=0 ) → Tensor bitwise_and ( ) bitwise_left_shift ( ) bitwise_not ( Tensor self ) → Tensor bitwise_or ( ) bitwise_right_shift ( ) bitwise_xor ( ) bmm ( Tensor self , Tensor mat2 ) → Tensor broadcast_to ( Tensor self , int[] size ) → Tensor cauchy_ ( Tensor(a!) self , float median=0.0 , float sigma=1.0) -> Tensor(a! ) ceil ( Tensor self ) → Tensor celu ( Tensor self , Scalar alpha=1.0 ) → Tensor channel_shuffle ( Tensor self , int groups ) → Tensor cholesky ( Tensor self , bool upper=False ) → Tensor cholesky_inverse ( Tensor self , bool upper=False ) → Tensor cholesky_solve ( Tensor self , Tensor input2 , bool upper=False ) → Tensor chunk ( Tensor self , int chunks , int dim=0 ) → Tensor[] clamp ( Tensor self , Scalar? min=None , Scalar? max=None ) → Tensor clamp_ ( Tensor(a!) self , Scalar? min=None , Scalar? max=None) -> Tensor(a! ) clip ( Tensor self , Scalar? min=None , Scalar? max=None ) → Tensor clone ( Tensor self , * , MemoryFormat? memory_format=None ) → Tensor cls_to_become alias of Parameter coalesce ( self : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase col_indices ( self : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase conj ( Tensor self ) → Tensor contiguous ( Tensor self , * , MemoryFormat memory_format=Contiguous ) → Tensor copy_ ( Tensor(a!) self , Tensor src , bool non_blocking=False) -> Tensor(a! ) cos ( Tensor self ) → Tensor cosh ( Tensor self ) → Tensor cpu ( ) Returns a copy of this object in CPU memory. If this object is already in CPU memory, then no copy is performed and the original object is returned. crow_indices ( self : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase cuda ( device = None , non_blocking = False ) Returns a copy of this object in CUDA memory. If this object is already in CUDA memory and on the correct device, then no copy is performed and the original object is returned. cummax ( Tensor self , int dim) -> (Tensor values , Tensor indices ) cummin ( Tensor self , int dim) -> (Tensor values , Tensor indices ) cumprod ( Tensor self , int dim , ScalarType? dtype=None ) → Tensor cumsum ( Tensor self , int dim=0 , ScalarType? dtype=None ) → Tensor data_ptr ( self : tensorplay._C.TensorBase ) → int deg2rad ( Tensor self ) → Tensor dense_dim ( self : tensorplay._C.TensorBase ) → int dequantize_per_channel ( Tensor self , Tensor scales , Tensor zero_points , int axis=0 ) → Tensor dequantize_per_tensor ( Tensor self , float scale , int zero_point ) → Tensor detach ( self : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase detach_ ( self : object ) → object diag ( Tensor self , int diagonal=0 ) → Tensor diag_embed ( Tensor self , int offset=0 , int dim1=0 , int dim2=1 ) → Tensor diagonal ( Tensor self , int offset=0 , int dim1=0 , int dim2=1 ) → Tensor diagonal_scatter ( Tensor self , Tensor src , int offset=0 , int dim1=0 , int dim2=1 ) → Tensor digamma ( Tensor self ) → Tensor dim ( self : tensorplay._C.TensorBase ) → int dist ( Tensor self , Tensor other , Scalar p=2 ) → Tensor div ( ) div_ ( ) div_ .Tensor(Tensor(a!) self, Tensor other) -> Tensor(a!) | div_ .Scalar(Tensor(a!) self, Scalar other) -> Tensor(a!) dot ( Tensor self , Tensor tensor ) → Tensor dsplit ( ) element_size ( self : tensorplay._C.TensorBase ) → int elu ( Tensor self , Scalar alpha=1 , Scalar scale=1 , Scalar input_scale=1 ) → Tensor eq ( ) equal ( Tensor self , Tensor other ) → bool erf ( Tensor self ) → Tensor erfc ( Tensor self ) → Tensor erfinv ( Tensor self ) → Tensor exp ( Tensor self ) → Tensor exp2 ( Tensor self ) → Tensor expand_as ( Tensor self , Tensor other ) → Tensor expm1 ( Tensor self ) → Tensor exponential_ ( Tensor(a!) self , float lambd=1.0) -> Tensor(a! ) fill_ ( ) fill_ .Scalar(Tensor(a!) self, Scalar value) -> Tensor(a!) fix ( Tensor self ) → Tensor flatten ( start_dim = 0 , end_dim = -1 ) Flattens a contiguous range of dims. floor ( Tensor self ) → Tensor frac ( Tensor self ) → Tensor gather ( Tensor self , int dim , Tensor index ) → Tensor gcd ( Tensor self , Tensor other ) → Tensor ge ( ) gelu ( Tensor self , str approximate="none" ) → Tensor geometric_ ( Tensor(a!) self , float p) -> Tensor(a! ) glu ( Tensor self , int dim=-1 ) → Tensor greater ( Tensor self , Tensor other ) → Tensor greater_equal ( Tensor self , Tensor other ) → Tensor gt ( ) hardshrink ( Tensor self , Scalar lambd=0.5 ) → Tensor hardsigmoid ( Tensor self ) → Tensor hardswish ( Tensor self ) → Tensor hardtanh ( Tensor self , Scalar min_val=-1 , Scalar max_val=1 ) → Tensor heaviside ( Tensor self , Tensor values ) → Tensor hsplit ( ) hypot ( Tensor self , Tensor other ) → Tensor i0e ( Tensor self ) → Tensor i1 ( Tensor self ) → Tensor i1e ( Tensor self ) → Tensor imag ( Tensor self ) → Tensor index_copy ( Tensor self , int dim , Tensor index , Tensor source ) → Tensor index_fill ( ) index_fill_ ( ) index_fill_ .Scalar(Tensor(a!) self, int dim, Tensor index, Scalar value) -> Tensor(a!) | index_fill_ .Tensor(Tensor(a!) self, int dim, Tensor index, Tensor value) -> Tensor(a!) index_put ( Tensor self , Tensor[] indices , Tensor values , bool accumulate=False ) → Tensor index_put_ ( Tensor(a!) self , Tensor[] indices , Tensor values , bool accumulate=False) -> Tensor(a! ) index_select ( Tensor self , int dim , Tensor index ) → Tensor inner ( Tensor self , Tensor other ) → Tensor is_channels_last ( self : tensorplay._C.TensorBase ) → bool is_channels_last_2d ( self : tensorplay._C.TensorBase ) → bool is_channels_last_3d ( self : tensorplay._C.TensorBase ) → bool is_coalesced ( self : tensorplay._C.TensorBase ) → bool is_complex ( self : tensorplay._C.TensorBase ) → bool is_contiguous ( * args , ** kwargs ) Overloaded function. is_contiguous(self: tensorplay._C.TensorBase) -> bool is_contiguous(self: tensorplay._C.TensorBase, memory_format: typing.SupportsInt | typing.SupportsIndex) -> bool is_floating_point ( self : tensorplay._C.TensorBase ) → bool is_pinned ( self : tensorplay._C.TensorBase ) → bool is_shared ( self : object ) → bool is_sparse_csr ( self : tensorplay._C.TensorBase ) → bool isclose ( Tensor self , Tensor other , float rtol=1e-05 , float atol=1e-08 , bool equal_nan=False ) → Tensor isfinite ( Tensor self ) → Tensor isinf ( Tensor self ) → Tensor isnan ( Tensor self ) → Tensor isneginf ( Tensor self ) → Tensor isposinf ( Tensor self ) → Tensor isreal ( Tensor self ) → Tensor itemsize ( self : tensorplay._C.TensorBase ) → int kthvalue ( Tensor self , int k , int dim=-1 , bool keepdim=False) -> (Tensor values , Tensor indices ) lcm ( Tensor self , Tensor other ) → Tensor le ( ) leaky_relu ( Tensor self , Scalar negative_slope=0.01 ) → Tensor lerp ( Tensor self , Tensor end , Scalar weight) -> Tensor | lerp.Tensor(Tensor self , Tensor end , Tensor weight ) → Tensor lerp_ ( ) lerp_ .Scalar(Tensor(a!) self, Tensor end, Scalar weight) -> Tensor(a!) | lerp_ .Tensor(Tensor(a!) self, Tensor end, Tensor weight) -> Tensor(a!) less ( Tensor self , Tensor other ) → Tensor less_equal ( Tensor self , Tensor other ) → Tensor lgamma ( Tensor self ) → Tensor log ( Tensor self ) → Tensor log10 ( Tensor self ) → Tensor log1p ( Tensor self ) → Tensor log2 ( Tensor self ) → Tensor log_normal_ ( Tensor(a!) self , float mean=1.0 , float std=2.0) -> Tensor(a! ) log_softmax ( Tensor self , int dim , ScalarType dtype=Undefined ) → Tensor logaddexp ( Tensor self , Tensor other ) → Tensor logaddexp2 ( Tensor self , Tensor other ) → Tensor logcumsumexp ( Tensor self , int dim , ScalarType? dtype=None ) → Tensor logical_not ( Tensor self ) → Tensor logical_or ( Tensor self , Tensor other ) → Tensor logical_xor ( Tensor self , Tensor other ) → Tensor logit ( Tensor self , Scalar? eps=None ) → Tensor logsumexp ( Tensor self , int dim , bool keepdim=False ) → Tensor lt ( ) masked_fill ( ) masked_fill_ ( ) masked_fill_ .Tensor(Tensor(a!) self, Tensor mask, Tensor value) -> Tensor(a!) | masked_fill_(Tensor(a!) self, Tensor mask, Scalar value) -> Tensor(a!) masked_scatter ( Tensor self , Tensor mask , Tensor source ) → Tensor masked_select ( Tensor self , Tensor mask ) → Tensor materialize ( shape , device = None , dtype = None ) Create a Parameter or Tensor with the same properties of the uninitialized one. Given a shape, it materializes a parameter in the same device and with the same dtype as the current one or the specified ones in the arguments. Parameters : shape – (tuple): the shape for the materialized tensor. device ( tensorplay.device ) – the desired device of the parameters and buffers in this module. Optional. dtype ( tensorplay.dtype ) – the desired floating point type of the floating point parameters and buffers in this module. Optional. matmul ( Tensor self , Tensor other ) → Tensor max ( Tensor self) -> Tensor | max.dim(Tensor self , int[] dim , bool keepdim=false ) → Tensor mean ( Tensor self , * , ScalarType dtype=Undefined) -> Tensor | mean.dim(Tensor self , int[] dim , bool keepdim=false , * , ScalarType dtype=Undefined ) → Tensor median ( Tensor self ) → Tensor memory_format ( self : tensorplay._C.TensorBase ) → int min ( Tensor self) -> Tensor | min.dim(Tensor self , int[] dim , bool keepdim=false ) → Tensor mish ( Tensor self ) → Tensor mm ( Tensor self , Tensor mat2 ) → Tensor mode ( Tensor self , int dim=-1 , bool keepdim=False) -> (Tensor values , Tensor indices ) modified_bessel_i1 ( Tensor self ) → Tensor modified_bessel_k0 ( Tensor self ) → Tensor modified_bessel_k1 ( Tensor self ) → Tensor moveaxis ( ) movedim ( Tensor self , int[] source , int[] destination ) → Tensor msort ( Tensor self ) → Tensor mul ( ) mul_ ( ) mul_ .Tensor(Tensor(a!) self, Tensor other) -> Tensor(a!) | mul_ .Scalar(Tensor(a!) self, Scalar other) -> Tensor(a!) mv ( Tensor self , Tensor vec ) → Tensor nan_to_num ( Tensor self , Scalar nan=0.0 , Scalar? posinf=None , Scalar? neginf=None ) → Tensor nanmean ( Tensor self , int? dim=None , bool keepdim=False , * , ScalarType? dtype=None ) → Tensor nanmedian ( Tensor self ) → Tensor nansum ( Tensor self , int[] dim=[] , bool keepdim=False ) → Tensor narrow ( Tensor self , int dim , int start , int length ) → Tensor ndimension ( ) → int Alias for dim() ne ( ) neg ( Tensor self ) → Tensor neg_ ( Tensor(a!) self) -> Tensor(a! ) negative ( Tensor self ) → Tensor nextafter ( Tensor self , Tensor other ) → Tensor norm ( Tensor self , float p=2.0) -> Tensor | norm.dim(Tensor self , int[] dim , float p=2.0 , bool keepdim=false ) → Tensor normal_ ( Tensor(a!) self , float mean=0.0 , float std=1.0) -> Tensor(a! ) not_equal ( Tensor self , Tensor other ) → Tensor numel ( self : tensorplay._C.TensorBase ) → int numpy ( self : object ) → numpy.ndarray outer ( Tensor self , Tensor vec2 ) → Tensor pdist ( Tensor self , float p=2.0 ) → Tensor pin_memory ( self : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase poisson ( Tensor self ) → Tensor polygamma ( int n , Tensor self ) → Tensor positive ( Tensor self ) → Tensor pow ( ) prelu ( Tensor self , Tensor weight ) → Tensor prod ( Tensor self , * , ScalarType dtype=Undefined) -> Tensor | prod.dim_IntList(Tensor self , int[] dim , bool keepdim=false , * , ScalarType dtype=Undefined ) → Tensor quantize_per_channel ( Tensor self , Tensor scales , Tensor zero_points , int axis=0 ) → Tensor quantize_per_tensor ( Tensor self , float scale , int zero_point , int quant_min=-128 , int quant_max=127 ) → Tensor rad2deg ( Tensor self ) → Tensor random_ ( Tensor(a!) self , int low=0 , int high=0) -> Tensor(a! ) ravel ( Tensor self ) → Tensor reciprocal ( Tensor self ) → Tensor register_hook ( hook ) Registers a backward hook (torch’s Tensor.register_hook ). The hook is called every time a gradient with respect to this tensor is computed. It may modify the gradient by returning a replacement Tensor; returning None leaves the gradient unchanged. Hooks compose in registration order. Returns a RemovableHandle whose remove() method (or context-manager form) unregisters the hook. register_post_accumulate_grad_hook ( hook ) Registers a hook (torch’s Tensor.register_post_accumulate_grad_hook ). The hook runs after the gradient has been accumulated into self.grad . It receives the tensor (the parameter) and its return value is ignored; unlike register_hook() it cannot replace the gradient, but it may modify self.grad in place. Only leaf tensors that require grad and are used in the autograd graph support this hook. Returns a RemovableHandle . relu ( Tensor self ) → Tensor relu6 ( Tensor self ) → Tensor relu_ ( Tensor(a!) self) -> Tensor(a! ) renorm ( Tensor self , Scalar p , int dim , Scalar maxnorm ) → Tensor repeat ( Tensor self , int[] repeats ) → Tensor requires_grad_ ( self : object , requires_grad : bool = True ) → object reshape ( Tensor self , int[] shape ) → Tensor resize_ ( Tensor(a!) self , int[] size) -> Tensor(a! ) retain_grad ( self : tensorplay._C.TensorBase ) → None rot90 ( Tensor self , int k=1 , int[] dims=[] ) → Tensor round ( Tensor self ) → Tensor rsqrt ( Tensor self ) → Tensor rsqrt_ ( Tensor(a!) self) -> Tensor(a! ) scaled_modified_bessel_k0 ( Tensor self ) → Tensor scaled_modified_bessel_k1 ( Tensor self ) → Tensor scatter ( ) scatter_ ( ) scatter_ .src(Tensor(a!) self, int dim, Tensor index, Tensor src) -> Tensor(a!) | scatter_ .value(Tensor(a!) self, int dim, Tensor index, Scalar value) -> Tensor(a!) scatter_add ( Tensor self , int dim , Tensor index , Tensor src ) → Tensor scatter_add_ ( Tensor(a!) self , int dim , Tensor index , Tensor src) -> Tensor(a! ) select ( Tensor self , int dim , int index ) → Tensor select_scatter ( Tensor self , Tensor src , int dim , int index ) → Tensor selu ( Tensor self ) → Tensor sgn ( Tensor self ) → Tensor sigmoid ( Tensor self ) → Tensor sign ( Tensor self ) → Tensor signbit ( Tensor self ) → Tensor silu ( Tensor self ) → Tensor sin ( Tensor self ) → Tensor sinc ( Tensor self ) → Tensor sinh ( Tensor self ) → Tensor size ( * args , ** kwargs ) Overloaded function. size(self: tensorplay._C.TensorBase) -> tensorplay._C.Size size(self: tensorplay._C.TensorBase, arg0: typing.SupportsInt | typing.SupportsIndex) -> int slice ( Tensor self , int dim=0 , int? start=None , int? end=None , int step=1 ) → Tensor slice_scatter ( Tensor self , Tensor src , int dim=0 , int? start=None , int? end=None , int step=1 ) → Tensor softmax ( Tensor self , int dim , ScalarType dtype=Undefined ) → Tensor softplus ( Tensor self , Scalar beta=1 , Scalar threshold=20 ) → Tensor softshrink ( Tensor self , Scalar lambd=0.5 ) → Tensor sort ( Tensor self , int dim=-1 , bool descending=False) -> (Tensor values , Tensor indices ) sparse_dim ( self : tensorplay._C.TensorBase ) → int sparse_mask ( self : tensorplay._C.TensorBase , mask : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase sparse_sum ( Tensor self , int[]? dim=None , ScalarType? dtype=None ) → Tensor spherical_bessel_j0 ( Tensor self ) → Tensor split ( Tensor self , int split_size , int dim=0) -> Tensor[] | split.sizes(Tensor self , int[] split_sizes , int dim=0 ) → Tensor[] split_with_sizes ( Tensor self , int[] split_sizes , int dim=0 ) → Tensor[] sqrt ( Tensor self ) → Tensor sqrt_ ( Tensor(a!) self) -> Tensor(a! ) square ( Tensor self ) → Tensor squeeze ( Tensor self) -> Tensor | squeeze.dim(Tensor self , int dim ) → Tensor std ( Tensor self , int correction=1) -> Tensor | std.dim(Tensor self , int[] dim , int correction=1 , bool keepdim=false ) → Tensor stride ( * args , ** kwargs ) Overloaded function. stride(self: tensorplay._C.TensorBase) -> tuple stride(self: tensorplay._C.TensorBase, arg0: typing.SupportsInt | typing.SupportsIndex) -> int sub ( ) sub_ ( ) sub_ .Tensor(Tensor(a!) self, Tensor other, * , Scalar alpha=1) -> Tensor(a!) | sub_ .Scalar(Tensor(a!) self, Scalar other, Scalar alpha=1) -> Tensor(a!) sum ( Tensor self , * , ScalarType dtype=Undefined) -> Tensor | sum.dim_IntList(Tensor self , int[] dim , bool keepdim=false , * , ScalarType dtype=Undefined ) → Tensor svd ( Tensor self , bool some=True , bool compute_uv=True) -> (Tensor U , Tensor S , Tensor V ) swapaxes ( Tensor self , int axis0 , int axis1 ) → Tensor swapdims ( Tensor self , int dim0 , int dim1 ) → Tensor t ( ) Returns the transpose of the tensor. Aliased to transpose(0, 1) to ensure correct autograd behavior (TransposeBackward). take ( Tensor self , Tensor index ) → Tensor take_along_dim ( Tensor self , Tensor indices , int? dim=None ) → Tensor tan ( Tensor self ) → Tensor tanh ( Tensor self ) → Tensor tensor_split ( ) tile ( Tensor self , int[] dims ) → Tensor to ( * args , ** kwargs ) Overloaded function. to(self: tensorplay._C.TensorBase, dtype: tensorplay._C.DType, non_blocking: bool = False, copy: bool = False) -> tensorplay._C.TensorBase to(self: tensorplay._C.TensorBase, device: tensorplay._C.Device, non_blocking: bool = False, copy: bool = False) -> tensorplay._C.TensorBase to(self: tensorplay._C.TensorBase, device: tensorplay._C.Device, dtype: tensorplay._C.DType, non_blocking: bool = False, copy: bool = False) -> tensorplay._C.TensorBase to_dense ( Tensor self ) → Tensor to_sparse ( Tensor self ) → Tensor to_sparse_csr ( Tensor self ) → Tensor tolist ( self : tensorplay._C.TensorBase ) → object trace ( Tensor self ) → Tensor transpose ( Tensor self , int dim0 , int dim1 ) → Tensor triangular_solve ( Tensor self , Tensor A , bool upper=False , bool transpose=False , bool unitriangular=False) -> (Tensor solution , Tensor cloned_coefficient ) tril ( Tensor self , int diagonal=0 ) → Tensor triu ( Tensor self , int diagonal=0 ) → Tensor trunc ( Tensor self ) → Tensor type ( dtype = None , non_blocking = False , ** kwargs ) Returns the type if dtype is not provided, else casts this object to the specified type. unbind ( Tensor self , int dim=0 ) → Tensor[] unflatten ( dim , sizes ) Expands a dimension of the input tensor over multiple dimensions. unfold ( dimension , size , step ) Returns a view of the original tensor which contains all slices of size size from self in the dimension dimension , stepping by step (torch’s Tensor.unfold ). Port of aten/src/ATen/native/TensorShape.cpp : the view appends a new trailing dimension of length size and re-strides dimension by step . uniform_ ( Tensor(a!) self , float from=0.0 , float to=1.0) -> Tensor(a! ) unsqueeze ( Tensor self , int dim ) → Tensor values ( self : tensorplay._C.TensorBase ) → tensorplay._C.TensorBase var ( Tensor self , int correction=1) -> Tensor | var.dim(Tensor self , int[] dim , int correction=1 , bool keepdim=false ) → Tensor vdot ( Tensor self , Tensor other ) → Tensor view ( Tensor self , int[] shape ) → Tensor vsplit ( ) zero_ ( Tensor(a!) self) -> Tensor(a! ) zeta ( Tensor self , Tensor other ) → Tensor # tensorplay.nn.utils.rnn.invert_permutation URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.utils.rnn.invert_permutation.html tensorplay.nn.utils.rnn.invert_permutation tensorplay.nn.utils.rnn. invert_permutation ( permutation : TensorBase | None ) → TensorBase | None [source] Returns the inverse of permutation . This is useful for converting between sorted and unsorted indices in a PackedSequence . Parameters : permutation ( Tensor , optional ) – a 1-D tensor of indices to invert # tensorplay.nn.utils.rnn.pack_padded_sequence URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.utils.rnn.pack_padded_sequence.html tensorplay.nn.utils.rnn.pack_padded_sequence tensorplay.nn.utils.rnn. pack_padded_sequence ( input : TensorBase , lengths , batch_first : bool = False , enforce_sorted : bool = True ) → PackedSequence [source] Packs a Tensor containing padded sequences of variable length. input can be of size T x B x * (if batch_first is False ) or B x T x * (if batch_first is True ) where T is the length of the longest sequence, B is the batch size, and * is any number of dimensions (including 0). For unsorted sequences, use enforce_sorted = False . If enforce_sorted is True , the sequences should be sorted by length in a decreasing order, i.e. input[:,0] should be the longest sequence, and input[:,B-1] the shortest one. enforce_sorted = True is only necessary for ONNX export. It is an inverse operation to pad_packed_sequence() , and hence pad_packed_sequence() can be used to recover the underlying tensor packed in PackedSequence . Note This function accepts any input that has at least two dimensions. You can apply it to pack the labels, and use the output of the RNN with them to compute the loss directly. A Tensor can be retrieved from a PackedSequence object by accessing its .data attribute. Parameters : input ( Tensor ) – padded batch of variable length sequences. lengths ( Tensor or list ( int ) ) – list of sequence lengths of each batch element (must be on the CPU if provided as a tensor). batch_first ( bool , optional ) – if True , the input is expected in B x T x * format, T x B x * otherwise. Default: False . enforce_sorted ( bool , optional ) – if True , the input is expected to contain sequences sorted by length in a decreasing order. If False , the input will get sorted unconditionally. Default: True . Warning The dim of input tensor will be truncated if its length larger than correspond value in length . Returns : a PackedSequence object # tensorplay.nn.utils.rnn.pack_sequence URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.utils.rnn.pack_sequence.html tensorplay.nn.utils.rnn.pack_sequence tensorplay.nn.utils.rnn. pack_sequence ( sequences , enforce_sorted : bool = True ) → PackedSequence [source] Packs a list of variable length Tensors. Consecutive call of the next functions: pad_sequence , pack_padded_sequence . sequences should be a list of Tensors of size L x * , where L is the length of a sequence and * is any number of trailing dimensions, including 0 . For unsorted sequences, use enforce_sorted = False . If enforce_sorted is True , the sequences should be sorted in the order of decreasing length. enforce_sorted = True is only necessary for ONNX export. Example >>> from tensorplay.nn.utils.rnn import pack_sequence >>> a = tp . tensor ([ 1 , 2 , 3 ]) >>> b = tp . tensor ([ 4 , 5 ]) >>> c = tp . tensor ([ 6 ]) >>> pack_sequence ([ a , b , c ]) PackedSequence(data=tensor([1, 4, 6, 2, 5, 3]), batch_sizes=tensor([3, 2, 1]), sorted_indices=None, unsorted_indices=None) Parameters : sequences ( list [ Tensor ] ) – A list of sequences of decreasing length. enforce_sorted ( bool , optional ) – if True , checks that the input contains sequences sorted by length in a decreasing order. If False , this condition is not checked. Default: True . Returns : a PackedSequence object # PackedSequence URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.utils.rnn.PackedSequence.html PackedSequence class tensorplay.nn.utils.rnn. PackedSequence ( data , batch_sizes = None , sorted_indices = None , unsorted_indices = None ) [source] Holds the data and list of batch_sizes of a packed sequence. All RNN modules accept packed sequences as inputs. Note Instances of this class should never be created manually. They are meant to be instantiated by functions like pack_padded_sequence() . Batch sizes represent the number elements at each sequence step in the batch, not the varying sequence lengths passed to pack_padded_sequence() . For instance, given data abc and x the PackedSequence would contain data axbc with batch_sizes=[2,1,1] . data Tensor containing packed sequence Type : Tensor batch_sizes Tensor of integers holding information about the batch size at each sequence step Type : Tensor sorted_indices Tensor of integers holding how this PackedSequence is constructed from sequences. Type : Tensor, optional unsorted_indices Tensor of integers holding how this to recover the original sequences with correct order. Type : Tensor, optional Note data can be on arbitrary device and of arbitrary dtype. sorted_indices and unsorted_indices must be int64 tensors on the same device as data . However, batch_sizes should always be a CPU int64 tensor. This invariant is maintained throughout PackedSequence class, and all functions that construct a PackedSequence in TensorPlay (i.e. they only pass in tensors conforming to this constraint). batch_sizes : TensorBase Alias for field number 1 count ( value , / ) Return number of occurrences of value. data : TensorBase Alias for field number 0 index ( value , start = 0 , stop = 9223372036854775807 , / ) Return first index of value. Raises ValueError if the value is not present. property is_cuda : bool Return true if self.data stored on a gpu. is_pinned ( ) → bool [source] Return true if self.data stored on in pinned memory. sorted_indices : TensorBase | None Alias for field number 2 to ( * args : Any , ** kwargs : Any ) [source] Perform dtype and/or device conversion on self.data . It has similar signature as tensorplay.Tensor.to() Note If the self.data Tensor already has the correct tensorplay.DType and tensorplay.Device , then self is returned. Otherwise, returns a copy with the desired configuration. unsorted_indices : TensorBase | None Alias for field number 3 # tensorplay.nn.utils.rnn.pad_packed_sequence URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.utils.rnn.pad_packed_sequence.html tensorplay.nn.utils.rnn.pad_packed_sequence tensorplay.nn.utils.rnn. pad_packed_sequence ( sequence : PackedSequence , batch_first : bool = False , padding_value : float = 0.0 , total_length : int | None = None ) [source] Pad a packed batch of variable length sequences. It is an inverse operation to pack_padded_sequence() . The returned Tensor’s data will be of size T x B x * (if batch_first is False ) or B x T x * (if batch_first is True ) , where T is the length of the longest sequence and B is the batch size. Example >>> from tensorplay.nn.utils.rnn import pack_padded_sequence , pad_packed_sequence >>> seq = tp . tensor ([[ 1 , 2 , 0 ], [ 3 , 0 , 0 ], [ 4 , 5 , 6 ]]) >>> lens = [ 2 , 1 , 3 ] >>> packed = pack_padded_sequence ( ... seq , lens , batch_first = True , enforce_sorted = False ... ) >>> packed PackedSequence(data=tensor([4, 1, 3, 5, 2, 6]), batch_sizes=tensor([3, 2, 1]), sorted_indices=tensor([2, 0, 1]), unsorted_indices=tensor([1, 2, 0])) >>> seq_unpacked , lens_unpacked = pad_packed_sequence ( packed , batch_first = True ) >>> seq_unpacked tensor([[1, 2, 0], [3, 0, 0], [4, 5, 6]]) >>> lens_unpacked tensor([2, 1, 3]) Note total_length is useful to implement the pack sequence -> recurrent network -> unpack sequence pattern in a model wrapped in DataParallel. Parameters : sequence ( PackedSequence ) – batch to pad batch_first ( bool , optional ) – if True , the output will be in B x T x * format, T x B x * otherwise. padding_value ( float , optional ) – values for padded elements. total_length ( int , optional ) – if not None , the output will be padded to have length total_length . This method will throw ValueError if total_length is less than the max sequence length in sequence . Returns : Tuple of Tensor containing the padded sequence, and a Tensor containing the list of lengths of each sequence in the batch. Batch elements will be re-ordered as they were ordered originally when the batch was passed to pack_padded_sequence() or pack_sequence() . # tensorplay.nn.utils.rnn.pad_sequence URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.utils.rnn.pad_sequence.html tensorplay.nn.utils.rnn.pad_sequence tensorplay.nn.utils.rnn. pad_sequence ( sequences , batch_first : bool = False , padding_value : float = 0.0 , padding_side : str = 'right' ) → TensorBase [source] Pad a list of variable length Tensors with padding_value . pad_sequence stacks a list of Tensors along a new dimension, and pads them to equal length. sequences can be list of sequences with size L x * , where L is length of the sequence and * is any number of dimensions (including 0 ). If batch_first is False , the output is of size T x B x * , and B x T x * otherwise, where B is the batch size (the number of elements in sequences` ), T is the length of the longest sequence. Example >>> from tensorplay.nn.utils.rnn import pad_sequence >>> a = tp . ones ( 25 , 300 ) >>> b = tp . ones ( 22 , 300 ) >>> c = tp . ones ( 15 , 300 ) >>> pad_sequence ([ a , b , c ]) . size () tensorplay.Size([25, 3, 300]) Note This function returns a Tensor of size T x B x * or B x T x * where T is the length of the longest sequence. This function assumes trailing dimensions and type of all the Tensors in sequences are same. Parameters : sequences ( list [ Tensor ] ) – list of variable length sequences. batch_first ( bool , optional ) – if True , the output will be in B x T x * format, T x B x * otherwise. Default: False . padding_value ( float , optional ) – value for padded elements. Default: 0 . padding_side ( str , optional ) – the side to pad the sequences on. Default: 'right' . Returns : Tensor of size T x B x * if batch_first is False . Tensor of size B x T x * otherwise # tensorplay.nn.utils.rnn.unpack_sequence URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.utils.rnn.unpack_sequence.html tensorplay.nn.utils.rnn.unpack_sequence tensorplay.nn.utils.rnn. unpack_sequence ( packed_sequences : PackedSequence ) [source] Unpack PackedSequence into a list of variable length Tensors. packed_sequences should be a PackedSequence object. Example >>> from tensorplay.nn.utils.rnn import pack_sequence , unpack_sequence >>> a = tp . tensor ([ 1 , 2 , 3 ]) >>> b = tp . tensor ([ 4 , 5 ]) >>> c = tp . tensor ([ 6 ]) >>> sequences = [ a , b , c ] >>> packed_sequences = pack_sequence ( sequences ) >>> unpacked_sequences = unpack_sequence ( packed_sequences ) Parameters : packed_sequences ( PackedSequence ) – A PackedSequence object. Returns : a list of Tensor objects # tensorplay.nn.utils.rnn.unpad_sequence URL: https://www.tensorplay.cn/docs/generated/tensorplay.nn.utils.rnn.unpad_sequence.html tensorplay.nn.utils.rnn.unpad_sequence tensorplay.nn.utils.rnn. unpad_sequence ( padded_sequences : TensorBase , lengths : TensorBase , batch_first : bool = False ) [source] Unpad padded Tensor into a list of variable length Tensors. unpad_sequence unstacks padded Tensor into a list of variable length Tensors. Example >>> from tensorplay.nn.utils.rnn import pad_sequence , unpad_sequence >>> a = tp . ones ( 25 , 300 ) >>> b = tp . ones ( 22 , 300 ) >>> c = tp . ones ( 15 , 300 ) >>> sequences = [ a , b , c ] >>> padded_sequences = pad_sequence ( sequences ) >>> lengths = tp . as_tensor ([ v . size ( 0 ) for v in sequences ]) >>> unpadded_sequences = unpad_sequence ( padded_sequences , lengths ) >>> tp . allclose ( sequences [ 0 ], unpadded_sequences [ 0 ]) True Parameters : padded_sequences ( Tensor ) – padded sequences. lengths ( Tensor ) – length of original (unpadded) sequences. batch_first ( bool , optional ) – whether batch dimension first or not. Default: False . Returns : a list of Tensor objects # tensorplay.normal_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.normal_.html tensorplay.normal_ tensorplay. normal_ ( input , mean = 0.0 , std = 1.0 ) [source] # tensorplay.one_hot URL: https://www.tensorplay.cn/docs/generated/tensorplay.one_hot.html tensorplay.one_hot tensorplay. one_hot ( input , num_classes = -1 ) [source] # Adadelta URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.adadelta.Adadelta.html Adadelta class tensorplay.optim.adadelta. Adadelta ( params , lr = 1.0 , rho = 0.9 , eps = 1e-06 , weight_decay = 0 , foreach = None , * , capturable = False , maximize = False , differentiable = False ) [source] Adadelta optimizer matching torch.optim.Adadelta . # Adafactor URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.Adafactor.html Adafactor class tensorplay.optim. Adafactor ( params , lr : float | TensorBase = 0.01 , beta2_decay : float = -0.8 , eps : tuple [ float | None , float ] = (None, 0.001) , d : float = 1.0 , weight_decay : float = 0.0 , * , foreach : bool | None = None , maximize : bool = False ) Implements Adafactor algorithm. \[ \begin{align}\begin{aligned}\begin{split}\begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma \text{(lr)}, \: \tau \text{(}\beta_2\text{ decay)}, \: \theta_0 \text{(params)}, \: f(\theta) \text{(objective)}, \\ &\hspace{15mm} \: \epsilon_1, \epsilon_2 \text{ (epsilons)}, \: d \text{(clipping threshold)}, \\ &\hspace{15mm} \: \lambda \text{(weight decay)}, \: \textit{maximize} \\ &\textbf{initialize} : \: R_0 \leftarrow 0 \text{ (second moment row factor)}, \\ &\hspace{23mm} \: C_0 \leftarrow 0 \text{ (second moment col factor)}, \\ &\hspace{23mm} \: \widehat{V}_0 \leftarrow 0 \text{ (second moment for vectors)} \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\\end{split}\\\begin{split} &\hspace{5mm}\textbf{if} \: \textit{maximize}: \\ &\hspace{10mm}G_t \leftarrow -\nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}\textbf{else} \\ &\hspace{10mm}G_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}\widehat{\beta}_{2_t} \leftarrow 1 - t^{\tau} \\ &\hspace{5mm}\rho_t \leftarrow min(\gamma, \frac{1}{\sqrt{t}}) \\ &\hspace{5mm}\alpha_t \leftarrow max(\epsilon_2, \text{RMS}(\theta_{t-1}))\rho_t \\ &\hspace{5mm}\theta_t \leftarrow \theta_{t-1} - \gamma \lambda \theta_{t-1} \\ &\hspace{5mm}\textbf{if} \: \text{dim}(G_t) > 1: \\ &\hspace{10mm}R_t \leftarrow \widehat{\beta}_{2_t}R_{t-1}+ (1-\widehat{\beta}_{2_t})(G_t \odot G_t) \cdot 1_m \\ &\hspace{10mm}C_t \leftarrow \widehat{\beta}_{2_t}C_{t-1}+ (1-\widehat{\beta}_{2_t}) 1^\top_n \cdot (G_t \odot G_t) \\ &\hspace{10mm}\widehat{V}_t \leftarrow \frac{R_t \cdot C_t}{max(1^\top_n \cdot R_t, \epsilon_1)} \\ &\hspace{5mm}\textbf{else} \\ &\hspace{10mm}\widehat{V}_t \leftarrow \widehat{\beta}_{2_t}\widehat{V}_{t-1}+ (1-\widehat{\beta}_{2_t}) \cdot (G_t \odot G_t) \\ &\hspace{5mm}U_t \leftarrow \frac{G_t}{max(\sqrt{\widehat{V}_t}, \epsilon_1)} \\ &\hspace{5mm}\widehat{U}_t \leftarrow \frac{U_t}{max(1, \frac{\text{RMS}(U_t)}{d})} \\ &\hspace{5mm}\theta_t \leftarrow \theta_{t-1} - \alpha_t \widehat{U}_t \\\end{split}\\\begin{split} &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned}\end{split}\end{aligned}\end{align} \] For further details regarding the algorithm we refer to Adafactor: Adaptive Learning Rates with Sublinear Memory Cost . Parameters : params lr ( float , Tensor , optional ) – unlike other optimizers, Adafactor does not require a learning rate, and Noam Shazeer and Mitchell Stern do not use lr at all. Deviating from the paper, this implementation uses lr for applying weight decay and as the maximum value for relative step size rho_t. Note that in the paper, a constant of 0.01 is used as the maximum value for relative step size, and so we set 0.01 as the default value. (default: 1e-2) beta2_decay ( float , optional ) – the decay rate of beta2. beta2 standardly refers to the coefficient used for computing the running average of the gradient squared. (default: -0.8) eps ( Tuple [ float , float ] , optional ) – epsilon1 is the term added to the denominator of the update calculation to improve numerical stability. This use of epsilon1 deviates from the algorithm written in the paper! See note below for more details. epsilon2 is the term used to avoid having too small a weight update when applying parameter scaling. (default: (None, 1e-3)) d ( float , optional ) – the clipping threshold, used to avoid larger-than-desired updates. weight_decay ( float , optional ) – weight decay coefficient (default: 1e-2) foreach ( bool , optional ) – whether foreach implementation of optimizer is used. Note that the foreach implementation uses ~ sizeof(params) more peak memory than the for-loop version due to the intermediates being a tensorlist vs just one tensor. As Adafactor is commonly used when memory is prohibitive, Adafactor will default to the slower single tensor for-loop implementation unless this flag is explicitly True. This behavior is contrary to other optimizers, which will attempt defaulting to foreach on CUDA for faster runtime. (default: None) maximize Note The implementation of Adafactor subtly differs from Noam Shazeer and Mitchell Stern and implementations in some other frameworks with its use of learning rate and \(\epsilon_1\) . Regarding the learning rate hyperparameter: Noam Shazeer and Mitchell Stern do not use lr at all, as the stated algorithm uses \(\rho_t\) and update clipping to affect the step size. This implementation allows lr to influence the maximum value for \(\rho_t\) : \[\begin{aligned} &\hspace{5mm}\rho_t \leftarrow min(\gamma, \frac{1}{\sqrt{t}}) \end{aligned}\] This differs from Noam Shazeer and Mitchell Stern, who use a constant of 0.01 as the maximum value of \(\rho_t\) \[\begin{aligned} &\hspace{5mm}\rho_t \leftarrow min(0.01, \frac{1}{\sqrt{t}}) \end{aligned}\] Noam Shazeer and Mitchell Stern do not enforce an opinion on how weight decay should be computed, and so we use the learning rate as a coefficient for decoupled weight decay, similar to what is suggested in Decoupled Weight Decay Regularization . Regarding the use of \(\epsilon_1\) : The implementation attempts to replicate the presumed intention of Noam Shazeer and Mitchell Stern to use \(\epsilon_1\) as a stabilizing term when the squared gradient becomes small. This stabilization can be written as \[\begin{split}\begin{aligned} &\hspace{5mm}R_t \leftarrow \widehat{\beta}_{2_t}R_{t-1}+ (1-\widehat{\beta}_{2_t})(G_t \odot G_t + 1_n \cdot 1^\top_m) \cdot 1_m \\ &\hspace{5mm}C_t \leftarrow \widehat{\beta}_{2_t}C_{t-1}+ (1-\widehat{\beta}_{2_t}) 1^\top_n \cdot (G_t \odot G_t + 1_n \cdot 1^\top_m) \\ &\hspace{5mm}\widehat{V}_t \leftarrow \frac{R_t \cdot C_t}{max(1^\top_n \cdot R_t, \epsilon_1)} \\ &\hspace{5mm}U_t \leftarrow \frac{G_t}{max(\sqrt{\widehat{V}_t}, \epsilon_1)} \\ \end{aligned}\end{split}\] where the row and column factors of gradient squared \(R_t\) and \(C_t\) are left alone, and we apply \(\epsilon_1\) at the final calculation of the variance estimate \(\widehat{V}_t\) and for the update \(U_t\) . This is in contrast to Noam Shazeer and Mitchell Stern and other frameworks which apply \(\epsilon_1\) to both row and column factors of the squared gradient, but not in the calculations after: \[\begin{split}\begin{aligned} &\hspace{5mm}R_t \leftarrow \widehat{\beta}_{2_t}R_{t-1}+ (1-\widehat{\beta}_{2_t})(G_t \odot G_t + \epsilon_1 1_n \cdot 1^\top_m) \cdot 1_m \\ &\hspace{5mm}C_t \leftarrow \widehat{\beta}_{2_t}C_{t-1}+ (1-\widehat{\beta}_{2_t}) 1^\top_n \cdot (G_t \odot G_t + \epsilon_1 1_n \cdot 1^\top_m) \\ &\hspace{5mm}\widehat{V}_t \leftarrow \frac{R_t \cdot C_t}{1^\top_n \cdot R_t} \\ &\hspace{5mm}U_t \leftarrow \frac{G_t}{\sqrt{\widehat{V}_t}} \\ \end{aligned}\end{split}\] You may note that Noam Shazeer and Mitchell Stern describe using the sum of squared gradients, while this implementation uses the mean instead. This choice is mathematically equivalent and allows for greater numerical stability for large sums. step ( closure = None ) [source] Perform a single optimization step. Parameters : closure ( Callable , optional ) – A closure that reevaluates the model and returns the loss. # Adagrad URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.adagrad.Adagrad.html Adagrad class tensorplay.optim.adagrad. Adagrad ( params : Iterable [ TensorBase ] | Iterable [ dict [ str , Any ] ] | Iterable [ tuple [ str , TensorBase ] ] , lr : float | TensorBase = 0.01 , lr_decay : float = 0 , weight_decay : float = 0 , initial_accumulator_value : float = 0 , eps : float = 1e-10 , foreach : bool | None = None , * , maximize : bool = False , differentiable : bool = False , fused : bool | None = None ) [source] share_memory ( ) → None [source] Calls tensor.share_memory_() on the state sum tensors. # Adam URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.adam.Adam.html Adam class tensorplay.optim.adam. Adam ( params , lr = 0.001 , betas = (0.9, 0.999) , eps = 1e-08 , weight_decay = 0 , amsgrad = False , * , foreach = None , maximize = False , capturable = False , differentiable = False , fused = None , decoupled_weight_decay = False ) [source] # Adamax URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.adamax.Adamax.html Adamax class tensorplay.optim.adamax. Adamax ( params , lr = 0.002 , betas = (0.9, 0.999) , eps = 1e-08 , weight_decay = 0 , foreach = None , * , maximize = False , differentiable = False , capturable = False ) [source] Adamax optimizer matching torch.optim.Adamax . # AdamW URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.adamw.AdamW.html AdamW class tensorplay.optim.adamw. AdamW ( params , lr = 0.001 , betas = (0.9, 0.999) , eps = 1e-08 , weight_decay = 0.01 , amsgrad = False , * , maximize = False , foreach = None , capturable = False , differentiable = False , fused = None ) [source] Adam with decoupled weight decay, matching torch.optim.AdamW . # ASGD URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.asgd.ASGD.html ASGD class tensorplay.optim.asgd. ASGD ( params , lr = 0.01 , lambd = 0.0001 , alpha = 0.75 , t0 = 1000000.0 , weight_decay = 0 , foreach = None , maximize = False , differentiable = False , capturable = False ) [source] Averaged stochastic gradient descent. # LBFGS URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.lbfgs.LBFGS.html LBFGS class tensorplay.optim.lbfgs. LBFGS ( params , lr = 1 , max_iter = 20 , max_eval = None , tolerance_grad = 1e-07 , tolerance_change = 1e-09 , history_size = 100 , line_search_fn = None , * , maximize = False ) [source] Limited-memory BFGS optimizer, aligned with torch.optim.LBFGS . # ChainedScheduler URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.lr_scheduler.ChainedScheduler.html ChainedScheduler class tensorplay.optim.lr_scheduler. ChainedScheduler ( schedulers : Sequence [ LRScheduler ] , optimizer : Optimizer | None = None ) [source] Chains a list of learning rate schedulers. Takes in a sequence of chainable learning rate schedulers and calls their step() functions consecutively in just one call to step(). Parameters : schedulers ( sequence ) – sequence of chained schedulers. optimizer ( Optimizer , optional ) – Wrapped optimizer. Default: None. Example >>> # xdoctest: +SKIP >>> # Assuming optimizer uses lr = 0.05 for all groups >>> # lr = 0.005 if epoch == 0 >>> # lr = 0.00450 if epoch == 1 >>> # lr = 0.00405 if epoch == 2 >>> # ... >>> # lr = 0.000675 if epoch == 19 >>> # lr = 0.006078 if epoch == 20 >>> # lr = 0.005470 if epoch == 21 >>> scheduler1 = ConstantLR ( optimizer , factor = 0.1 , total_iters = 20 ) >>> scheduler2 = ExponentialLR ( optimizer , gamma = 0.9 ) >>> scheduler = ChainedScheduler ([ scheduler1 , scheduler2 ], optimizer = optimizer ) >>> for epoch in range ( 100 ): >>> train ( ... ) >>> validate ( ... ) >>> scheduler . step () get_last_lr ( ) → list [ float | TensorBase ] Get the most recent learning rates computed by this scheduler. Returns : A list of learning rates with entries for each of the optimizer’s param_groups , with the same types as their group["lr"] s. Return type : list [ float | Tensor] Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. get_lr ( ) → list [ float | TensorBase ] Compute the next learning rate for each of the optimizer’s param_groups . Returns : A list of learning rates for each of the optimizer’s param_groups with the same types as their current group["lr"] s. Return type : list [ float | Tensor] Note If you’re trying to inspect the most recent learning rate, use get_last_lr() instead. Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. load_state_dict ( state_dict : dict [ str , Any ] ) → None [source] Load the scheduler’s state. Parameters : state_dict ( dict ) – scheduler state. Should be an object returned from a call to state_dict() . state_dict ( ) → dict [ str , Any ] [source] Return the state of the scheduler as a dict . It contains an entry for every variable in self.__dict__ which is not the optimizer. The wrapped scheduler states will also be saved. step ( ) → None [source] Perform a step. # ConstantLR URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.lr_scheduler.ConstantLR.html ConstantLR class tensorplay.optim.lr_scheduler. ConstantLR ( optimizer : Optimizer , factor : float = 0.3333333333333333 , total_iters : int = 5 , last_epoch : int = -1 ) [source] Multiply the learning rate of each parameter group by a small constant factor. The multiplication is done until the number of epoch reaches a pre-defined milestone: total_iters. Notice that such multiplication of the small constant factor can happen simultaneously with other changes to the learning rate from outside this scheduler. When last_epoch=-1, sets initial lr as lr. Parameters : optimizer ( Optimizer ) – Wrapped optimizer. factor ( float ) – The number we multiply learning rate until the milestone. Default: 1./3. total_iters ( int ) – The number of steps that the scheduler multiplies the learning rate by the factor. Default: 5. last_epoch ( int ) – The index of the last epoch. Default: -1. Example >>> # xdoctest: +SKIP >>> # Assuming optimizer uses lr = 0.05 for all groups >>> # lr = 0.025 if epoch == 0 >>> # lr = 0.025 if epoch == 1 >>> # lr = 0.025 if epoch == 2 >>> # lr = 0.025 if epoch == 3 >>> # ... >>> # lr = 0.05 if epoch >= 40 >>> scheduler = ConstantLR ( optimizer , factor = 0.5 , total_iters = 40 ) >>> for epoch in range ( 100 ): >>> train ( ... ) >>> validate ( ... ) >>> scheduler . step () get_last_lr ( ) → list [ float | TensorBase ] Get the most recent learning rates computed by this scheduler. Returns : A list of learning rates with entries for each of the optimizer’s param_groups , with the same types as their group["lr"] s. Return type : list [ float | Tensor] Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. get_lr ( ) → list [ float | TensorBase ] [source] Compute the next learning rate for each of the optimizer’s param_groups . When last_epoch is 0, this method scales the group["lr"] s in each of the optimizer’s param_groups by factor . Once total_iters is reached, it undoes this, scaling by 1 / factor . Returns : A list of learning rates for each of the optimizer’s param_groups with the same types as their current group["lr"] s. Return type : list [ float | Tensor] Note If you’re trying to inspect the most recent learning rate, use get_last_lr() instead. Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. load_state_dict ( state_dict : dict [ str , Any ] ) → None Load the scheduler’s state. Parameters : state_dict ( dict ) – scheduler state. Should be an object returned from a call to state_dict() . state_dict ( ) → dict [ str , Any ] Return the state of the scheduler as a dict . It contains an entry for every variable in self.__dict__ which is not the optimizer. step ( epoch : int | None = None ) → None Step the scheduler. Parameters : epoch ( int , optional ) – Deprecated since version 1.4: If provided, sets last_epoch to epoch and uses _get_closed_form_lr() if it is available. This is not universally supported. Use step() without arguments instead. Note Call this method after calling the optimizer’s step() . # CosineAnnealingLR URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.lr_scheduler.CosineAnnealingLR.html CosineAnnealingLR class tensorplay.optim.lr_scheduler. CosineAnnealingLR ( optimizer : Optimizer , T_max : int , eta_min : float = 0.0 , last_epoch : int = -1 ) [source] Set the learning rate of each parameter group using a cosine annealing schedule. The learning rate is updated recursively using: \[\eta_{t+1} = \eta_{\min} + (\eta_t - \eta_{\min}) \cdot \frac{1 + \cos\left(\frac{(T_{cur}+1) \pi}{T_{max}}\right)} {1 + \cos\left(\frac{T_{cur} \pi}{T_{max}}\right)}\] This implements a recursive approximation of the closed-form schedule proposed in SGDR: Stochastic Gradient Descent with Warm Restarts : \[\eta_t = \eta_{\min} + \frac{1}{2}(\eta_{\max} - \eta_{\min}) \left( 1 + \cos\left(\frac{T_{cur} \pi}{T_{max}}\right) \right)\] where: \(\eta_t\) is the learning rate at step \(t\) \(T_{cur}\) is the number of epochs since the last restart \(T_{max}\) is the maximum number of epochs in a cycle Note Although SGDR includes periodic restarts, this implementation performs cosine annealing without restarts , so \(T_{cur} = t\) and increases monotonically with each call to step() . Parameters : optimizer ( Optimizer ) – Wrapped optimizer. T_max ( int ) – Maximum number of iterations. eta_min ( float ) – Minimum learning rate. Default: 0. last_epoch ( int ) – The index of the last epoch. Default: -1. Example >>> # xdoctest: +SKIP >>> num_epochs = 100 >>> scheduler = CosineAnnealingLR ( optimizer , T_max = num_epochs ) >>> for epoch in range ( num_epochs ): >>> train ( ... ) >>> validate ( ... ) >>> scheduler . step () get_last_lr ( ) → list [ float | TensorBase ] Get the most recent learning rates computed by this scheduler. Returns : A list of learning rates with entries for each of the optimizer’s param_groups , with the same types as their group["lr"] s. Return type : list [ float | Tensor] Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. get_lr ( ) → list [ float | TensorBase ] [source] Compute the next learning rate for each of the optimizer’s param_groups . Scales the group["lr"] s in the optimizer’s param_groups such that their learning rates approximate \[\texttt{eta\_min} + \frac{1}{2} (\texttt{base\_lr} - \texttt{eta\_min}) \left(1 + \cos\left(\pi \cdot \frac{\texttt{last\_epoch}}{\texttt{T\_max}}\right) \right)\] Returns : A list of learning rates for each of the optimizer’s param_groups with the same types as their current group["lr"] s. Return type : list [ float | Tensor] Note If you’re trying to inspect the most recent learning rate, use get_last_lr() instead. Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. load_state_dict ( state_dict : dict [ str , Any ] ) → None Load the scheduler’s state. Parameters : state_dict ( dict ) – scheduler state. Should be an object returned from a call to state_dict() . state_dict ( ) → dict [ str , Any ] Return the state of the scheduler as a dict . It contains an entry for every variable in self.__dict__ which is not the optimizer. step ( epoch : int | None = None ) → None Step the scheduler. Parameters : epoch ( int , optional ) – Deprecated since version 1.4: If provided, sets last_epoch to epoch and uses _get_closed_form_lr() if it is available. This is not universally supported. Use step() without arguments instead. Note Call this method after calling the optimizer’s step() . # CosineAnnealingWarmRestarts URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.lr_scheduler.CosineAnnealingWarmRestarts.html CosineAnnealingWarmRestarts class tensorplay.optim.lr_scheduler. CosineAnnealingWarmRestarts ( optimizer : Optimizer , T_0 : int , T_mult : int = 1 , eta_min : float = 0.0 , last_epoch : int = -1 ) [source] Set the learning rate of each parameter group using a cosine annealing schedule. The \(\eta_{max}\) is set to the initial lr, \(T_{cur}\) is the number of epochs since the last restart and \(T_{i}\) is the number of epochs between two warm restarts in SGDR: \[\eta_t = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min})\left(1 + \cos\left(\frac{T_{cur}}{T_{i}}\pi\right)\right)\] When \(T_{cur}=T_{i}\) , set \(\eta_t = \eta_{min}\) . When \(T_{cur}=0\) after restart, set \(\eta_t=\eta_{max}\) . It has been proposed in SGDR: Stochastic Gradient Descent with Warm Restarts . Parameters : optimizer ( Optimizer ) – Wrapped optimizer. T_0 ( int ) – Number of iterations until the first restart. T_mult ( int , optional ) – A factor by which \(T_{i}\) increases after a restart. Default: 1. eta_min ( float , optional ) – Minimum learning rate. Default: 0. last_epoch ( int , optional ) – The index of the last epoch. Default: -1. Example >>> # xdoctest: +SKIP >>> optimizer = torch . optim . SGD ( model . parameters (), lr = 0.05 ) >>> scheduler = torch . optim . lr_scheduler . CosineAnnealingWarmRestarts ( ... optimizer , T_0 = 20 ... ) >>> for epoch in range ( 100 ): >>> train ( ... ) >>> validate ( ... ) >>> scheduler . step () get_last_lr ( ) → list [ float | TensorBase ] Get the most recent learning rates computed by this scheduler. Returns : A list of learning rates with entries for each of the optimizer’s param_groups , with the same types as their group["lr"] s. Return type : list [ float | Tensor] Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. get_lr ( ) → list [ float | TensorBase ] [source] Compute the next learning rate for each of the optimizer’s param_groups . Computes learning rates for the optimizer’s param_groups following: \[\texttt{eta\_min} + \frac{1}{2}(\texttt{base\_lr} - \texttt{eta\_min})\left(1 + \cos\left(\pi \cdot \frac{\texttt{T\_cur}}{\texttt{T\_i}}\right)\right)\] Where T_cur is the number of epochs since the last restart and T_i is the number of epochs between two restarts. Both T_cur and T_i are updated in step() , and T_i becomes T_mult times larger after each restart. Returns : A list of learning rates for each of the optimizer’s param_groups with the same types as their current group["lr"] s. Return type : list [ float | Tensor] Note If you’re trying to inspect the most recent learning rate, use get_last_lr() instead. Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. load_state_dict ( state_dict : dict [ str , Any ] ) → None Load the scheduler’s state. Parameters : state_dict ( dict ) – scheduler state. Should be an object returned from a call to state_dict() . state_dict ( ) → dict [ str , Any ] Return the state of the scheduler as a dict . It contains an entry for every variable in self.__dict__ which is not the optimizer. step ( epoch = None ) → None [source] Step could be called after every batch update. Example >>> # xdoctest: +SKIP("Undefined vars") >>> scheduler = CosineAnnealingWarmRestarts ( optimizer , T_0 , T_mult ) >>> iters = len ( dataloader ) >>> for epoch in range ( 20 ): >>> for i , sample in enumerate ( dataloader ): >>> inputs , labels = sample [ 'inputs' ], sample [ 'labels' ] >>> optimizer . zero_grad () >>> outputs = net ( inputs ) >>> loss = criterion ( outputs , labels ) >>> loss . backward () >>> optimizer . step () >>> scheduler . step ( epoch + i / iters ) This function can be called in an interleaved way. Example >>> # xdoctest: +SKIP("Undefined vars") >>> scheduler = CosineAnnealingWarmRestarts ( optimizer , T_0 , T_mult ) >>> for epoch in range ( 20 ): >>> scheduler . step () >>> scheduler . step ( 26 ) >>> scheduler . step () # scheduler.step(27), instead of scheduler(20) # CyclicLR URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.lr_scheduler.CyclicLR.html CyclicLR class tensorplay.optim.lr_scheduler. CyclicLR ( optimizer : Optimizer , base_lr : float | list [ float ] , max_lr : float | list [ float ] , step_size_up : int = 2000 , step_size_down : int | None = None , mode : Literal [ 'triangular' , 'triangular2' , 'exp_range' ] = 'triangular' , gamma : float = 1.0 , scale_fn : Callable [ [ float ] , float ] | None = None , scale_mode : Literal [ 'cycle' , 'iterations' ] = 'cycle' , cycle_momentum : bool = True , base_momentum : float = 0.8 , max_momentum : float = 0.9 , last_epoch : int = -1 ) [source] Sets the learning rate of each parameter group according to cyclical learning rate policy (CLR). The policy cycles the learning rate between two boundaries with a constant frequency, as detailed in the paper Cyclical Learning Rates for Training Neural Networks . The distance between the two boundaries can be scaled on a per-iteration or per-cycle basis. Cyclical learning rate policy changes the learning rate after every batch. step should be called after a batch has been used for training. This class has three built-in policies, as put forth in the paper: “triangular”: A basic triangular cycle without amplitude scaling. “triangular2”: A basic triangular cycle that scales initial amplitude by half each cycle. “exp_range”: A cycle that scales initial amplitude by \(\text{gamma}^{\text{cycle iterations}}\) at each cycle iteration. This implementation was adapted from the github repo: bckenstler/CLR Parameters : optimizer ( Optimizer ) – Wrapped optimizer. base_lr ( float or list ) – Initial learning rate which is the lower boundary in the cycle for each parameter group. max_lr ( float or list ) – Upper learning rate boundaries in the cycle for each parameter group. Functionally, it defines the cycle amplitude (max_lr - base_lr). The lr at any cycle is the sum of base_lr and some scaling of the amplitude; therefore max_lr may not actually be reached depending on scaling function. step_size_up ( int ) – Number of training iterations in the increasing half of a cycle. Default: 2000 step_size_down ( int ) – Number of training iterations in the decreasing half of a cycle. If step_size_down is None, it is set to step_size_up. Default: None mode ( str ) – One of {triangular, triangular2, exp_range}. Values correspond to policies detailed above. If scale_fn is not None, this argument is ignored. Default: ‘triangular’ gamma ( float ) – Constant in ‘exp_range’ scaling function: gamma**(cycle iterations) Default: 1.0 scale_fn ( function ) – Custom scaling policy defined by a single argument lambda function, where 0 <= scale_fn(x) <= 1 for all x >= 0. If specified, then ‘mode’ is ignored. Default: None scale_mode ( str ) – {‘cycle’, ‘iterations’}. Defines whether scale_fn is evaluated on cycle number or cycle iterations (training iterations since start of cycle). Default: ‘cycle’ cycle_momentum ( bool ) – If True , momentum is cycled inversely to learning rate between ‘base_momentum’ and ‘max_momentum’. Default: True base_momentum ( float or list ) – Lower momentum boundaries in the cycle for each parameter group. Note that momentum is cycled inversely to learning rate; at the peak of a cycle, momentum is ‘base_momentum’ and learning rate is ‘max_lr’. Default: 0.8 max_momentum ( float or list ) – Upper momentum boundaries in the cycle for each parameter group. Functionally, it defines the cycle amplitude (max_momentum - base_momentum). The momentum at any cycle is the difference of max_momentum and some scaling of the amplitude; therefore base_momentum may not actually be reached depending on scaling function. Note that momentum is cycled inversely to learning rate; at the start of a cycle, momentum is ‘max_momentum’ and learning rate is ‘base_lr’ Default: 0.9 last_epoch ( int ) – The index of the last batch. This parameter is used when resuming a training job. Since step() should be invoked after each batch instead of after each epoch, this number represents the total number of batches computed, not the total number of epochs computed. When last_epoch=-1, the schedule is started from the beginning. Default: -1 Example >>> # xdoctest: +SKIP >>> optimizer = torch . optim . SGD ( model . parameters (), lr = 0.1 , momentum = 0.9 ) >>> scheduler = torch . optim . lr_scheduler . CyclicLR ( ... optimizer , ... base_lr = 0.01 , ... max_lr = 0.1 , ... step_size_up = 10 , ... ) >>> data_loader = torch . utils . data . DataLoader ( ... ) >>> for epoch in range ( 10 ): >>> for batch in data_loader : >>> train_batch ( ... ) >>> scheduler . step () get_last_lr ( ) → list [ float | TensorBase ] Get the most recent learning rates computed by this scheduler. Returns : A list of learning rates with entries for each of the optimizer’s param_groups , with the same types as their group["lr"] s. Return type : list [ float | Tensor] Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. get_lr ( ) → list [ float | TensorBase ] [source] Compute the next learning rate for each of the optimizer’s param_groups . Advances each group["lr"] in the optimizer’s param_groups along a cycle between the group’s base_lr and max_lr using scale_fn() . Returns : A list of learning rates for each of the optimizer’s param_groups with the same types as their current group["lr"] s. Return type : list [ float | Tensor] Note If you’re trying to inspect the most recent learning rate, use get_last_lr() instead. Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. Note This method treats last_epoch as the index of the previous batch. Note When cycle_momentum is True , this method has a side effect of updating the optimizer’s momentum. load_state_dict ( state_dict : dict [ str , Any ] ) → None [source] Load the scheduler’s state. scale_fn ( x ) → float [source] Get the scaling policy. state_dict ( ) → dict [ str , Any ] [source] Return the state of the scheduler as a dict . It contains an entry for every variable in self.__dict__ which is not the optimizer. The learning rate lambda functions will only be saved if they are callable objects and not if they are functions or lambdas. When saving or loading the scheduler, please make sure to also save or load the state of the optimizer. step ( epoch : int | None = None ) → None Step the scheduler. Parameters : epoch ( int , optional ) – Deprecated since version 1.4: If provided, sets last_epoch to epoch and uses _get_closed_form_lr() if it is available. This is not universally supported. Use step() without arguments instead. Note Call this method after calling the optimizer’s step() . # ExponentialLR URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.lr_scheduler.ExponentialLR.html ExponentialLR class tensorplay.optim.lr_scheduler. ExponentialLR ( optimizer : Optimizer , gamma : float , last_epoch : int = -1 ) [source] Decays the learning rate of each parameter group by gamma every epoch. When last_epoch=-1, sets initial lr as lr. Parameters : optimizer ( Optimizer ) – Wrapped optimizer. gamma ( float ) – Multiplicative factor of learning rate decay. last_epoch ( int ) – The index of last epoch. Default: -1. Example >>> # xdoctest: +SKIP >>> scheduler = ExponentialLR ( optimizer , gamma = 0.95 ) >>> for epoch in range ( 100 ): >>> train ( ... ) >>> validate ( ... ) >>> scheduler . step () get_last_lr ( ) → list [ float | TensorBase ] Get the most recent learning rates computed by this scheduler. Returns : A list of learning rates with entries for each of the optimizer’s param_groups , with the same types as their group["lr"] s. Return type : list [ float | Tensor] Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. get_lr ( ) → list [ float | TensorBase ] [source] Compute the next learning rate for each of the optimizer’s param_groups . Multiplies the current group["lr"] s in the optimizer’s param_groups by gamma . Returns : A list of learning rates for each of the optimizer’s param_groups with the same types as their current group["lr"] s. Return type : list [ float | Tensor] Note If you’re trying to inspect the most recent learning rate, use get_last_lr() instead. Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. load_state_dict ( state_dict : dict [ str , Any ] ) → None Load the scheduler’s state. Parameters : state_dict ( dict ) – scheduler state. Should be an object returned from a call to state_dict() . state_dict ( ) → dict [ str , Any ] Return the state of the scheduler as a dict . It contains an entry for every variable in self.__dict__ which is not the optimizer. step ( epoch : int | None = None ) → None Step the scheduler. Parameters : epoch ( int , optional ) – Deprecated since version 1.4: If provided, sets last_epoch to epoch and uses _get_closed_form_lr() if it is available. This is not universally supported. Use step() without arguments instead. Note Call this method after calling the optimizer’s step() . # LambdaLR URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.lr_scheduler.LambdaLR.html LambdaLR class tensorplay.optim.lr_scheduler. LambdaLR ( optimizer : Optimizer , lr_lambda : Callable [ [ int ] , float ] | list [ Callable [ [ int ] , float ] ] , last_epoch : int = -1 ) [source] Sets the initial learning rate. The learning rate of each parameter group is set to the initial lr times a given function. When last_epoch=-1, sets initial lr as lr. Parameters : optimizer ( Optimizer ) – Wrapped optimizer. lr_lambda ( function or list ) – A function which computes a multiplicative factor given an integer parameter epoch, or a list of such functions, one for each group in optimizer.param_groups. last_epoch ( int ) – The index of last epoch. Default: -1. Example >>> # xdoctest: +SKIP >>> # Assuming optimizer has two groups. >>> num_epochs = 100 >>> lambda1 = lambda epoch : epoch // 30 >>> lambda2 = lambda epoch : 0.95 ** epoch >>> scheduler = LambdaLR ( optimizer , lr_lambda = [ lambda1 , lambda2 ]) >>> for epoch in range ( num_epochs ): >>> train ( ... ) >>> validate ( ... ) >>> scheduler . step () >>> >>> # Alternatively, you can use a single lambda function for all groups. >>> scheduler = LambdaLR ( opt , lr_lambda = lambda epoch : epoch // 30 ) >>> for epoch in range ( num_epochs ): >>> train ( ... ) >>> validate ( ... ) >>> scheduler . step () get_last_lr ( ) → list [ float | TensorBase ] Get the most recent learning rates computed by this scheduler. Returns : A list of learning rates with entries for each of the optimizer’s param_groups , with the same types as their group["lr"] s. Return type : list [ float | Tensor] Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. get_lr ( ) → list [ float | TensorBase ] [source] Compute the next learning rate for each of the optimizer’s param_groups . Scales the base_lrs by the outputs of the lr_lambdas at last_epoch . Returns : A list of learning rates for each of the optimizer’s param_groups with the same types as their current group["lr"] s. Return type : list [ float | Tensor] Note If you’re trying to inspect the most recent learning rate, use get_last_lr() instead. Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. load_state_dict ( state_dict : dict [ str , Any ] ) → None [source] Load the scheduler’s state. When saving or loading the scheduler, please make sure to also save or load the state of the optimizer. Parameters : state_dict ( dict ) – scheduler state. Should be an object returned from a call to state_dict() . state_dict ( ) → dict [ str , Any ] [source] Return the state of the scheduler as a dict . It contains an entry for every variable in self.__dict__ which is not the optimizer. The learning rate lambda functions will only be saved if they are callable objects and not if they are functions or lambdas. When saving or loading the scheduler, please make sure to also save or load the state of the optimizer. step ( epoch : int | None = None ) → None Step the scheduler. Parameters : epoch ( int , optional ) – Deprecated since version 1.4: If provided, sets last_epoch to epoch and uses _get_closed_form_lr() if it is available. This is not universally supported. Use step() without arguments instead. Note Call this method after calling the optimizer’s step() . # LinearLR URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.lr_scheduler.LinearLR.html LinearLR class tensorplay.optim.lr_scheduler. LinearLR ( optimizer : Optimizer , start_factor : float = 0.3333333333333333 , end_factor : float = 1.0 , total_iters : int = 5 , last_epoch : int = -1 ) [source] Decays the learning rate of each parameter group by linearly changing small multiplicative factor. The multiplication is done until the number of epoch reaches a pre-defined milestone: total_iters. Notice that such decay can happen simultaneously with other changes to the learning rate from outside this scheduler. When last_epoch=-1, sets initial lr as lr. Parameters : optimizer ( Optimizer ) – Wrapped optimizer. start_factor ( float ) – The number we multiply learning rate in the first epoch. The multiplication factor changes towards end_factor in the following epochs. Default: 1./3. end_factor ( float ) – The number we multiply learning rate at the end of linear changing process. Default: 1.0. total_iters ( int ) – The number of iterations that multiplicative factor reaches to 1. Default: 5. last_epoch ( int ) – The index of the last epoch. Default: -1. Example >>> # xdoctest: +SKIP >>> # Assuming optimizer uses lr = 0.05 for all groups >>> # lr = 0.003687 if epoch == 0 >>> # lr = 0.004875 if epoch == 1 >>> # lr = 0.006062 if epoch == 2 >>> # lr = 0.00725 if epoch == 3 >>> # ... >>> # lr = 0.05 if epoch >= 40 >>> scheduler = LinearLR ( optimizer , start_factor = 0.05 , total_iters = 40 ) >>> for epoch in range ( 100 ): >>> train ( ... ) >>> validate ( ... ) >>> scheduler . step () get_last_lr ( ) → list [ float | TensorBase ] Get the most recent learning rates computed by this scheduler. Returns : A list of learning rates with entries for each of the optimizer’s param_groups , with the same types as their group["lr"] s. Return type : list [ float | Tensor] Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. get_lr ( ) → list [ float | TensorBase ] [source] Compute the next learning rate for each of the optimizer’s param_groups . Scales the group["lr"] s in the optimizer’s param_groups such that successive steps interpolate linearly from start_factor up to end_factor across total_iters steps. Returns : A list of learning rates for each of the optimizer’s param_groups with the same types as their current group["lr"] s. Return type : list [ float | Tensor] Note If you’re trying to inspect the most recent learning rate, use get_last_lr() instead. Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. load_state_dict ( state_dict : dict [ str , Any ] ) → None Load the scheduler’s state. Parameters : state_dict ( dict ) – scheduler state. Should be an object returned from a call to state_dict() . state_dict ( ) → dict [ str , Any ] Return the state of the scheduler as a dict . It contains an entry for every variable in self.__dict__ which is not the optimizer. step ( epoch : int | None = None ) → None Step the scheduler. Parameters : epoch ( int , optional ) – Deprecated since version 1.4: If provided, sets last_epoch to epoch and uses _get_closed_form_lr() if it is available. This is not universally supported. Use step() without arguments instead. Note Call this method after calling the optimizer’s step() . # LRScheduler URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.lr_scheduler.LRScheduler.html LRScheduler class tensorplay.optim.lr_scheduler. LRScheduler ( optimizer : Optimizer , last_epoch : int = -1 ) [source] Base class for all learning rate schedulers. Subclasses implement get_lr() and optionally override step() to define scheduling behavior. Parameters : optimizer ( Optimizer ) – The optimizer this scheduler will adjust the learning rates of. last_epoch ( int ) – Index of the last epoch seen by the scheduler. Use -1 (default) to initialize the scheduler. Only use a non-default value when restoring this scheduler from a saved checkpoint. Warning Initializing a scheduler overwrites its optimizer’s param_group["lr"] s. When restoring a checkpoint, initialize the scheduler before calling your optimizer’s load_state_dict() to avoid overwriting the loaded learning rates. get_last_lr ( ) → list [ float | TensorBase ] [source] Get the most recent learning rates computed by this scheduler. Returns : A list of learning rates with entries for each of the optimizer’s param_groups , with the same types as their group["lr"] s. Return type : list [ float | Tensor] Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. get_lr ( ) → list [ float | TensorBase ] [source] Compute the next learning rate for each of the optimizer’s param_groups . Returns : A list of learning rates for each of the optimizer’s param_groups with the same types as their current group["lr"] s. Return type : list [ float | Tensor] Note If you’re trying to inspect the most recent learning rate, use get_last_lr() instead. Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. load_state_dict ( state_dict : dict [ str , Any ] ) → None [source] Load the scheduler’s state. Parameters : state_dict ( dict ) – scheduler state. Should be an object returned from a call to state_dict() . state_dict ( ) → dict [ str , Any ] [source] Return the state of the scheduler as a dict . It contains an entry for every variable in self.__dict__ which is not the optimizer. step ( epoch : int | None = None ) → None [source] Step the scheduler. Parameters : epoch ( int , optional ) – Deprecated since version 1.4: If provided, sets last_epoch to epoch and uses _get_closed_form_lr() if it is available. This is not universally supported. Use step() without arguments instead. Note Call this method after calling the optimizer’s step() . # MultiplicativeLR URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.lr_scheduler.MultiplicativeLR.html MultiplicativeLR class tensorplay.optim.lr_scheduler. MultiplicativeLR ( optimizer : Optimizer , lr_lambda : Callable [ [ int ] , float ] | list [ Callable [ [ int ] , float ] ] , last_epoch : int = -1 ) [source] Multiply the learning rate of each parameter group by the factor given in the specified function. When last_epoch=-1, set initial lr as lr. Parameters : optimizer ( Optimizer ) – Wrapped optimizer. lr_lambda ( function or list ) – A function which computes a multiplicative factor given an integer parameter epoch, or a list of such functions, one for each group in optimizer.param_groups. last_epoch ( int ) – The index of last epoch. Default: -1. Example >>> # xdoctest: +SKIP >>> lmbda = lambda epoch : 0.95 >>> scheduler = MultiplicativeLR ( optimizer , lr_lambda = lmbda ) >>> for epoch in range ( 100 ): >>> train ( ... ) >>> validate ( ... ) >>> scheduler . step () get_last_lr ( ) → list [ float | TensorBase ] Get the most recent learning rates computed by this scheduler. Returns : A list of learning rates with entries for each of the optimizer’s param_groups , with the same types as their group["lr"] s. Return type : list [ float | Tensor] Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. get_lr ( ) → list [ float | TensorBase ] [source] Compute the next learning rate for each of the optimizer’s param_groups . Scales the current group["lr"] s in each of the optimizer’s param_groups by the outputs of the lr_lambdas at last_epoch . Returns : A list of learning rates for each of the optimizer’s param_groups with the same types as their current group["lr"] s. Return type : list [ float | Tensor] Note If you’re trying to inspect the most recent learning rate, use get_last_lr() instead. Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. load_state_dict ( state_dict : dict [ str , Any ] ) → None [source] Load the scheduler’s state. Parameters : state_dict ( dict ) – scheduler state. Should be an object returned from a call to state_dict() . state_dict ( ) → dict [ str , Any ] [source] Return the state of the scheduler as a dict . It contains an entry for every variable in self.__dict__ which is not the optimizer. The learning rate lambda functions will only be saved if they are callable objects and not if they are functions or lambdas. step ( epoch : int | None = None ) → None Step the scheduler. Parameters : epoch ( int , optional ) – Deprecated since version 1.4: If provided, sets last_epoch to epoch and uses _get_closed_form_lr() if it is available. This is not universally supported. Use step() without arguments instead. Note Call this method after calling the optimizer’s step() . # MultiStepLR URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.lr_scheduler.MultiStepLR.html MultiStepLR class tensorplay.optim.lr_scheduler. MultiStepLR ( optimizer : Optimizer , milestones : Iterable [ int ] , gamma : float = 0.1 , last_epoch : int = -1 ) [source] Decays the learning rate of each parameter group by gamma once the number of epoch reaches one of the milestones. Notice that such decay can happen simultaneously with other changes to the learning rate from outside this scheduler. When last_epoch=-1, sets initial lr as lr. Parameters : optimizer ( Optimizer ) – Wrapped optimizer. milestones ( list ) – List of epoch indices. Must be increasing. gamma ( float ) – Multiplicative factor of learning rate decay. Default: 0.1. last_epoch ( int ) – The index of last epoch. Default: -1. Example >>> # xdoctest: +SKIP >>> # Assuming optimizer uses lr = 0.05 for all groups >>> # lr = 0.05 if epoch < 30 >>> # lr = 0.005 if 30 <= epoch < 80 >>> # lr = 0.0005 if epoch >= 80 >>> scheduler = MultiStepLR ( optimizer , milestones = [ 30 , 80 ], gamma = 0.1 ) >>> for epoch in range ( 100 ): >>> train ( ... ) >>> validate ( ... ) >>> scheduler . step () get_last_lr ( ) → list [ float | TensorBase ] Get the most recent learning rates computed by this scheduler. Returns : A list of learning rates with entries for each of the optimizer’s param_groups , with the same types as their group["lr"] s. Return type : list [ float | Tensor] Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. get_lr ( ) → list [ float | TensorBase ] [source] Compute the next learning rate for each of the optimizer’s param_groups . If the current epoch is in milestones , decays the group["lr"] s in the optimizer’s param_groups by gamma . Returns : A list of learning rates for each of the optimizer’s param_groups with the same types as their current group["lr"] s. Return type : list [ float | Tensor] Note If you’re trying to inspect the most recent learning rate, use get_last_lr() instead. Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. Note If the current epoch appears in milestones n times, we scale by gamma to the power of n load_state_dict ( state_dict : dict [ str , Any ] ) → None Load the scheduler’s state. Parameters : state_dict ( dict ) – scheduler state. Should be an object returned from a call to state_dict() . state_dict ( ) → dict [ str , Any ] Return the state of the scheduler as a dict . It contains an entry for every variable in self.__dict__ which is not the optimizer. step ( epoch : int | None = None ) → None Step the scheduler. Parameters : epoch ( int , optional ) – Deprecated since version 1.4: If provided, sets last_epoch to epoch and uses _get_closed_form_lr() if it is available. This is not universally supported. Use step() without arguments instead. Note Call this method after calling the optimizer’s step() . # OneCycleLR URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.lr_scheduler.OneCycleLR.html OneCycleLR class tensorplay.optim.lr_scheduler. OneCycleLR ( optimizer : Optimizer , max_lr : float | list [ float ] , total_steps : int | None = None , epochs : int | None = None , steps_per_epoch : int | None = None , pct_start : float = 0.3 , anneal_strategy : Literal [ 'cos' , 'linear' ] = 'cos' , cycle_momentum : bool = True , base_momentum : float | list [ float ] = 0.85 , max_momentum : float | list [ float ] = 0.95 , div_factor : float = 25.0 , final_div_factor : float = 10000.0 , three_phase : bool = False , last_epoch : int = -1 ) [source] Sets the learning rate of each parameter group according to the 1cycle learning rate policy. The 1cycle policy anneals the learning rate from an initial learning rate to some maximum learning rate and then from that maximum learning rate to some minimum learning rate much lower than the initial learning rate. This policy was initially described in the paper Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates . The 1cycle learning rate policy changes the learning rate after every batch. step should be called after a batch has been used for training. This scheduler is not chainable. Note also that the total number of steps in the cycle can be determined in one of two ways (listed in order of precedence): A value for total_steps is explicitly provided. A number of epochs (epochs) and a number of steps per epoch (steps_per_epoch) are provided. In this case, the number of total steps is inferred by total_steps = epochs * steps_per_epoch You must either provide a value for total_steps or provide a value for both epochs and steps_per_epoch. The default behaviour of this scheduler follows the fastai implementation of 1cycle, which claims that “unpublished work has shown even better results by using only two phases”. To mimic the behaviour of the original paper instead, set three_phase=True . Parameters : optimizer ( Optimizer ) – Wrapped optimizer. max_lr ( float or list ) – Upper learning rate boundaries in the cycle for each parameter group. total_steps ( int ) – The total number of steps in the cycle. Note that if a value is not provided here, then it must be inferred by providing a value for epochs and steps_per_epoch. Default: None epochs ( int ) – The number of epochs to train for. This is used along with steps_per_epoch in order to infer the total number of steps in the cycle if a value for total_steps is not provided. Default: None steps_per_epoch ( int ) – The number of steps per epoch to train for. This is used along with epochs in order to infer the total number of steps in the cycle if a value for total_steps is not provided. Default: None pct_start ( float ) – The percentage of the cycle (in number of steps) spent increasing the learning rate. Default: 0.3 anneal_strategy ( str ) – {‘cos’, ‘linear’} Specifies the annealing strategy: “cos” for cosine annealing, “linear” for linear annealing. Default: ‘cos’ cycle_momentum ( bool ) – If True , momentum is cycled inversely to learning rate between ‘base_momentum’ and ‘max_momentum’. Default: True base_momentum ( float or list ) – Lower momentum boundaries in the cycle for each parameter group. Note that momentum is cycled inversely to learning rate; at the peak of a cycle, momentum is ‘base_momentum’ and learning rate is ‘max_lr’. Default: 0.85 max_momentum ( float or list ) – Upper momentum boundaries in the cycle for each parameter group. Functionally, it defines the cycle amplitude (max_momentum - base_momentum). Note that momentum is cycled inversely to learning rate; at the start of a cycle, momentum is ‘max_momentum’ and learning rate is ‘base_lr’ Default: 0.95 div_factor ( float ) – Determines the initial learning rate via initial_lr = max_lr/div_factor Default: 25 final_div_factor ( float ) – Determines the minimum learning rate via min_lr = initial_lr/final_div_factor Default: 1e4 three_phase ( bool ) – If True , use a third phase of the schedule to annihilate the learning rate according to ‘final_div_factor’ instead of modifying the second phase (the first two phases will be symmetrical about the step indicated by ‘pct_start’). last_epoch ( int ) – The index of the last batch. This parameter is used when resuming a training job. Since step() should be invoked after each batch instead of after each epoch, this number represents the total number of batches computed, not the total number of epochs computed. When last_epoch=-1, the schedule is started from the beginning. Default: -1 Example >>> # xdoctest: +SKIP >>> data_loader = torch . utils . data . DataLoader ( ... ) >>> optimizer = torch . optim . SGD ( model . parameters (), lr = 1e-4 , momentum = 0.9 ) >>> scheduler = torch . optim . lr_scheduler . OneCycleLR ( ... optimizer , max_lr = 0.01 , steps_per_epoch = len ( data_loader ), epochs = 10 ... ) >>> for epoch in range ( 10 ): >>> for batch in data_loader : >>> train_batch ( ... ) >>> optimizer . step () >>> scheduler . step () get_last_lr ( ) → list [ float | TensorBase ] Get the most recent learning rates computed by this scheduler. Returns : A list of learning rates with entries for each of the optimizer’s param_groups , with the same types as their group["lr"] s. Return type : list [ float | Tensor] Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. get_lr ( ) → list [ float | TensorBase ] [source] Compute the next learning rate for each of the optimizer’s param_groups . Finds the appropriate _schedule_phases entry for the current step and interpolates between its start_lr and end_lr using _anneal_func() . Returns : A list of learning rates for each of the optimizer’s param_groups with the same types as their current group["lr"] s. Return type : list [ float | Tensor] Note If you’re trying to inspect the most recent learning rate, use get_last_lr() instead. Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. Note When cycle_momentum is True , this method has a side effect of updating the optimizer’s momentum. load_state_dict ( state_dict : dict [ str , Any ] ) → None Load the scheduler’s state. Parameters : state_dict ( dict ) – scheduler state. Should be an object returned from a call to state_dict() . state_dict ( ) → dict [ str , Any ] Return the state of the scheduler as a dict . It contains an entry for every variable in self.__dict__ which is not the optimizer. step ( epoch : int | None = None ) → None Step the scheduler. Parameters : epoch ( int , optional ) – Deprecated since version 1.4: If provided, sets last_epoch to epoch and uses _get_closed_form_lr() if it is available. This is not universally supported. Use step() without arguments instead. Note Call this method after calling the optimizer’s step() . # PolynomialLR URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.lr_scheduler.PolynomialLR.html PolynomialLR class tensorplay.optim.lr_scheduler. PolynomialLR ( optimizer : Optimizer , total_iters : int = 5 , power : float = 1.0 , last_epoch : int = -1 ) [source] Decays the learning rate of each parameter group using a polynomial function in the given total_iters. When last_epoch=-1, sets initial lr as lr. Parameters : optimizer ( Optimizer ) – Wrapped optimizer. total_iters ( int ) – The number of steps that the scheduler decays the learning rate. Default: 5. power ( float ) – The power of the polynomial. Default: 1.0. Example >>> # xdoctest: +SKIP("undefined vars") >>> # Assuming optimizer uses lr = 0.05 for all groups >>> # lr = 0.0490 if epoch == 0 >>> # lr = 0.0481 if epoch == 1 >>> # lr = 0.0472 if epoch == 2 >>> # ... >>> # lr = 0.0 if epoch >= 50 >>> scheduler = PolynomialLR ( optimizer , total_iters = 50 , power = 0.9 ) >>> for epoch in range ( 100 ): >>> train ( ... ) >>> validate ( ... ) >>> scheduler . step () get_last_lr ( ) → list [ float | TensorBase ] Get the most recent learning rates computed by this scheduler. Returns : A list of learning rates with entries for each of the optimizer’s param_groups , with the same types as their group["lr"] s. Return type : list [ float | Tensor] Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. get_lr ( ) → list [ float | TensorBase ] [source] Compute the next learning rate for each of the optimizer’s param_groups . Scales the group["lr"] s in the optimizer’s param_groups such that the learning rates follow \[\texttt{base\_lr} \cdot \left(1 - \frac{\texttt{last\_epoch}} {\texttt{total\_iters}} \right)^\texttt{power}\] Returns the current learning rates unchanged after total_iters is reached. Returns : A list of learning rates for each of the optimizer’s param_groups with the same types as their current group["lr"] s. Return type : list [ float | Tensor] Note If you’re trying to inspect the most recent learning rate, use get_last_lr() instead. Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. load_state_dict ( state_dict : dict [ str , Any ] ) → None Load the scheduler’s state. Parameters : state_dict ( dict ) – scheduler state. Should be an object returned from a call to state_dict() . state_dict ( ) → dict [ str , Any ] Return the state of the scheduler as a dict . It contains an entry for every variable in self.__dict__ which is not the optimizer. step ( epoch : int | None = None ) → None Step the scheduler. Parameters : epoch ( int , optional ) – Deprecated since version 1.4: If provided, sets last_epoch to epoch and uses _get_closed_form_lr() if it is available. This is not universally supported. Use step() without arguments instead. Note Call this method after calling the optimizer’s step() . # ReduceLROnPlateau URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.lr_scheduler.ReduceLROnPlateau.html ReduceLROnPlateau class tensorplay.optim.lr_scheduler. ReduceLROnPlateau ( optimizer : Optimizer , mode : Literal [ 'min' , 'max' ] = 'min' , factor : float = 0.1 , patience : int = 10 , threshold : float = 0.0001 , threshold_mode : Literal [ 'rel' , 'abs' ] = 'rel' , cooldown : int = 0 , min_lr : list [ float ] | float = 0 , eps : float = 1e-08 ) [source] Reduce learning rate when a metric has stopped improving. Models often benefit from reducing the learning rate by a factor of 2-10 once learning stagnates. This scheduler reads a metrics quantity and if no improvement is seen for a ‘patience’ number of epochs, the learning rate is reduced. Parameters : optimizer ( Optimizer ) – Wrapped optimizer. mode ( str ) – One of min , max . In min mode, lr will be reduced when the quantity monitored has stopped decreasing; in max mode it will be reduced when the quantity monitored has stopped increasing. Default: ‘min’. factor ( float ) – Factor by which the learning rate will be reduced. new_lr = lr * factor. Default: 0.1. patience ( int ) – The number of allowed epochs with no improvement after which the learning rate will be reduced. For example, consider the case of having no patience ( patience = 0 ). In the first epoch, a baseline is established and is always considered good as there’s no previous baseline. In the second epoch, if the performance is worse than the baseline, we have what is considered an intolerable epoch. Since the count of intolerable epochs (1) is greater than the patience level (0), the learning rate is reduced at the end of this epoch. From the third epoch onwards, the learning rate continues to be reduced at the end of each epoch if the performance is worse than the baseline. If the performance improves or remains the same, the learning rate is not adjusted. Default: 10. threshold ( float ) – Threshold for measuring the new optimum, to only focus on significant changes. Default: 1e-4. threshold_mode ( str ) – One of rel , abs . In rel mode, the dynamic threshold is computed as: best * (1 + threshold) if mode == ‘max’ best * (1 - threshold) if mode == ‘min’ In abs mode, the dynamic threshold is computed as: best + threshold if mode == ‘max’ best - threshold if mode == ‘min’ Default: ‘rel’. cooldown ( int ) – Number of epochs to wait before resuming normal operation after lr has been reduced. Default: 0. min_lr ( float or list ) – A scalar or a list of scalars. A lower bound on the learning rate of all param groups or each group respectively. Default: 0. eps ( float ) – Minimal decay applied to lr. If the difference between new and old lr is smaller than eps, the update is ignored. Default: 1e-8. Example >>> # xdoctest: +SKIP >>> optimizer = torch . optim . SGD ( model . parameters (), lr = 0.1 , momentum = 0.9 ) >>> scheduler = ReduceLROnPlateau ( optimizer , "min" ) >>> for epoch in range ( 10 ): >>> train ( ... ) >>> val_loss = validate ( ... ) >>> # Note that step should be called after validate() >>> scheduler . step ( val_loss ) get_last_lr ( ) → list [ float | TensorBase ] Get the most recent learning rates computed by this scheduler. Returns : A list of learning rates with entries for each of the optimizer’s param_groups , with the same types as their group["lr"] s. Return type : list [ float | Tensor] Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. get_lr ( ) → list [ float | TensorBase ] Compute the next learning rate for each of the optimizer’s param_groups . Returns : A list of learning rates for each of the optimizer’s param_groups with the same types as their current group["lr"] s. Return type : list [ float | Tensor] Note If you’re trying to inspect the most recent learning rate, use get_last_lr() instead. Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. load_state_dict ( state_dict : dict [ str , Any ] ) → None [source] Load the scheduler’s state. state_dict ( ) → dict [ str , Any ] Return the state of the scheduler as a dict . It contains an entry for every variable in self.__dict__ which is not the optimizer. step ( metrics : SupportsFloat , epoch = None ) → None [source] Perform a step. # SequentialLR URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.lr_scheduler.SequentialLR.html SequentialLR class tensorplay.optim.lr_scheduler. SequentialLR ( optimizer : Optimizer , schedulers : list [ LRScheduler ] , milestones : list [ int ] , last_epoch : int = -1 ) [source] Contains a list of schedulers expected to be called sequentially during the optimization process. Specifically, the schedulers will be called according to the milestone points, which should provide exact intervals by which each scheduler should be called at a given epoch. Parameters : optimizer ( Optimizer ) – Wrapped optimizer. schedulers ( list ) – List of chained schedulers. milestones ( list ) – List of integers that reflects milestone points. last_epoch ( int ) – The index of last epoch. Default: -1. Example >>> # xdoctest: +SKIP >>> # Assuming optimizer uses lr = 0.05 for all groups >>> # lr = 0.005 if epoch == 0 >>> # lr = 0.005 if epoch == 1 >>> # lr = 0.005 if epoch == 2 >>> # ... >>> # lr = 0.05 if epoch == 20 >>> # lr = 0.045 if epoch == 21 >>> # lr = 0.0405 if epoch == 22 >>> scheduler1 = ConstantLR ( optimizer , factor = 0.1 , total_iters = 20 ) >>> scheduler2 = ExponentialLR ( optimizer , gamma = 0.9 ) >>> scheduler = SequentialLR ( ... optimizer , ... schedulers = [ scheduler1 , scheduler2 ], ... milestones = [ 20 ], ... ) >>> for epoch in range ( 100 ): >>> train ( ... ) >>> validate ( ... ) >>> scheduler . step () get_last_lr ( ) → list [ float | TensorBase ] Get the most recent learning rates computed by this scheduler. Returns : A list of learning rates with entries for each of the optimizer’s param_groups , with the same types as their group["lr"] s. Return type : list [ float | Tensor] Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. get_lr ( ) → list [ float | TensorBase ] Compute the next learning rate for each of the optimizer’s param_groups . Returns : A list of learning rates for each of the optimizer’s param_groups with the same types as their current group["lr"] s. Return type : list [ float | Tensor] Note If you’re trying to inspect the most recent learning rate, use get_last_lr() instead. Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. load_state_dict ( state_dict : dict [ str , Any ] ) → None [source] Load the scheduler’s state. Parameters : state_dict ( dict ) – scheduler state. Should be an object returned from a call to state_dict() . recursive_undo ( sched = None ) → None [source] Recursively undo any step performed by the initialization of schedulers. state_dict ( ) → dict [ str , Any ] [source] Return the state of the scheduler as a dict . It contains an entry for every variable in self.__dict__ which is not the optimizer. The wrapped scheduler states will also be saved. step ( ) → None [source] Perform a step. # StepLR URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.lr_scheduler.StepLR.html StepLR class tensorplay.optim.lr_scheduler. StepLR ( optimizer : Optimizer , step_size : int , gamma : float = 0.1 , last_epoch : int = -1 ) [source] Decays the learning rate of each parameter group by gamma every step_size epochs. Notice that such decay can happen simultaneously with other changes to the learning rate from outside this scheduler. When last_epoch=-1, sets initial lr as lr. Parameters : optimizer ( Optimizer ) – Wrapped optimizer. step_size ( int ) – Period of learning rate decay. gamma ( float ) – Multiplicative factor of learning rate decay. Default: 0.1. last_epoch ( int ) – The index of last epoch. Default: -1. Example >>> # xdoctest: +SKIP >>> # Assuming optimizer uses lr = 0.05 for all groups >>> # lr = 0.05 if epoch < 30 >>> # lr = 0.005 if 30 <= epoch < 60 >>> # lr = 0.0005 if 60 <= epoch < 90 >>> # ... >>> scheduler = StepLR ( optimizer , step_size = 30 , gamma = 0.1 ) >>> for epoch in range ( 100 ): >>> train ( ... ) >>> validate ( ... ) >>> scheduler . step () get_last_lr ( ) → list [ float | TensorBase ] Get the most recent learning rates computed by this scheduler. Returns : A list of learning rates with entries for each of the optimizer’s param_groups , with the same types as their group["lr"] s. Return type : list [ float | Tensor] Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. get_lr ( ) → list [ float | TensorBase ] [source] Compute the next learning rate for each of the optimizer’s param_groups . If the current epoch is a non-zero multiple of step_size , we scale the current group["lr"] s in the optimizer’s param_groups by gamma . Returns : A list of learning rates for each of the optimizer’s param_groups with the same types as their current group["lr"] s. Return type : list [ float | Tensor] Note If you’re trying to inspect the most recent learning rate, use get_last_lr() instead. Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. load_state_dict ( state_dict : dict [ str , Any ] ) → None Load the scheduler’s state. Parameters : state_dict ( dict ) – scheduler state. Should be an object returned from a call to state_dict() . state_dict ( ) → dict [ str , Any ] Return the state of the scheduler as a dict . It contains an entry for every variable in self.__dict__ which is not the optimizer. step ( epoch : int | None = None ) → None Step the scheduler. Parameters : epoch ( int , optional ) – Deprecated since version 1.4: If provided, sets last_epoch to epoch and uses _get_closed_form_lr() if it is available. This is not universally supported. Use step() without arguments instead. Note Call this method after calling the optimizer’s step() . # Muon URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.Muon.html Muon class tensorplay.optim. Muon ( params , lr : float = 0.001 , weight_decay : float = 0.1 , momentum : float = 0.95 , nesterov : bool = True , ns_coefficients : tuple [ float , float , float ] = (3.4445, -4.775, 2.0315) , eps : float = 1e-07 , ns_steps : int = 5 , adjust_lr_fn : str | None = None ) Implements Muon algorithm. \[ \begin{align}\begin{aligned}\begin{split}\begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma \text{ (lr)},\ \lambda \text{ (weight decay)},\ \mu \text{ (momentum)},\ \textit{nesterov}\in\{True,False\},\\ &\hspace{13mm}(a,b,c)\ \text{ (NS coefficients)},\ \varepsilon \text{ (epsilon)},\ k \text{ (NS steps)},\ \theta_0 \text{ (params)},\ f(\theta) \text{ (objective)} \\ &\textbf{initialize} : B_0 \leftarrow 0 \text{ (momentum buffer)} \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for}\ t=1\ \textbf{to}\ \ldots\ \textbf{do} \\[0.25ex] &\hspace{5mm} g_t \leftarrow \nabla_{\theta} f_t(\theta_{t-1}) \\[0.25ex] &\hspace{5mm} B_t \leftarrow \mu B_{t-1} + g_t \\[0.25ex] &\hspace{5mm} \widetilde{B}_t \leftarrow \begin{cases} g_t + \mu B_t, & \text{if nesterov}=True \\ B_t, & \text{if nesterov}=False \end{cases} \\[1.0ex] &\hspace{5mm} O_t \leftarrow \mathrm{NS}^{(a,b,c)}_{k}\!\big(\widetilde{B}_t;\ \varepsilon\big) \\[0.5ex] &\hspace{5mm} \theta_t \leftarrow \theta_{t-1} - \gamma\,\lambda\,\theta_{t-1} \quad\text{(decoupled weight decay)} \\[0.25ex]\end{split}\\\begin{split} &\hspace{5mm} \gamma \leftarrow \mathrm{AdjustLR}\!\big(\gamma;\ \mathrm{shape}\!\big(\theta_t \big) \big) \\[0.25ex] &\hspace{5mm} \theta_t \leftarrow \theta_t - \gamma\, O_t \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\mathbf{return}\ \theta_t \\[-1.ex] &\rule{110mm}{0.4pt}s \end{aligned}\end{split}\end{aligned}\end{align} \] Here, \(\mathrm{NS}^{(a,b,c)}_{k}(\cdot;\varepsilon)\) denotes \(k\) iterations of the Newton–Schulz orthogonalization operator parameterized by coefficients \((a,b,c)\) with numerical stabilization \(\varepsilon\) . The purpose for \(\mathrm{AdjustLR}\!\big(\gamma;\ \mathrm{shape}\!\big(\theta_t \big) \big)\) is to make the orthogonalized update scale consistently across rectangular matrices. Keller’s original implementation scales the update by \(\sqrt{\max\!\left(1, \frac{A}{B}\right)}\) , where \(A\) and \(B\) are dimensions of the matrix being optimized, which represent fan-out and fan-in for a Linear weight matrix. Moonshot’s implementation focuses on matching \(RMS\) of AdamW. The adjustment is computed as: \(\gamma \leftarrow {0.2}\gamma\,\sqrt{\max\!\left({A}, {B}\right)}\) The method is adopted from Muon is Scalable for LLM Training . Research results show that with this adjustment Muon can directly reuse the learning rate and weight decay tuned for AdamW. Jeremy Bernstein in Deriving Muon proposes a scaling condition on the spectral norm, which scales the update by \(\sqrt{\frac{A}{B}}\) . This is similar to the Keller’s “original” implementation but removes clamping down to 1. We provide these options for the learning rate adjustment: “original”, which follows Keller’s implementation, “match_rms_adamw”, which refers to Moonshot’s implementation, and “spectral_unclamped”, which matches Bernstein’s implementation. If adjust_lr_fn is not specified, the default is “original”. For further details regarding the algorithm we refer to Muon: An optimizer for hidden layers in neural networks , Muon is Scalable for LLM Training , and Deriving Muon . Parameters : Other ( params. Note that Muon is an optimizer for 2D parameters of neural network hidden layers. ) – parameters, such as bias, and embedding, should be optimized by a standard method such as AdamW. lr ( float , Tensor , optional ) – learning rate (default: 1e-3). weight_decay ( float , optional ) – weight decay (L2 penalty). (default: 0.1) momentum ( float , optional ) – momentum factor (default: 0.95) nesterov ( bool , optional ) – enables Nesterov momentum. Only applicable when momentum is non-zero ns_coefficients ( tuple of three floats , optional ) – coefficients (a,b,c) for the Newton–Schulz orthogonalization polynomial (default: (3.4445, -4.775, 2.0315)) eps ( float , optional ) – term added to the denominator for numerical stability. (default: 1e-07) ns_steps ( int , optional ) – number of Newton–Schulz iteration steps. (default: 5) adjust_lr_fn ( str , optional ) – function to adjust learning rate. One of “original”, “match_rms_adamw”, and “spectral_unclamped”. If not specified, we will default to use “original”. (default: None) Example >>> # xdoctest: +SKIP >>> # Muon only supports 2D params; use a standard optimizer >>> # such as AdamW for biases, embeddings, and other non-2D >>> # parameters. >>> muon_params = [ ... p for p in model . parameters () if p . ndim == 2 ... ] >>> other_params = [ ... p for p in model . parameters () if p . ndim != 2 ... ] >>> optim_muon = torch . optim . Muon ( ... muon_params , lr = 0.02 , momentum = 0.95 ... ) >>> optim_adamw = torch . optim . AdamW ( ... other_params , lr = 3e-4 , weight_decay = 0.01 ... ) >>> optim_muon . zero_grad () >>> optim_adamw . zero_grad () >>> loss_fn ( model ( input ), target ) . backward () >>> optim_muon . step () >>> optim_adamw . step () step ( closure = None ) [source] Performs a single optimization step. # NAdam URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.nadam.NAdam.html NAdam class tensorplay.optim.nadam. NAdam ( params , lr = 0.002 , betas = (0.9, 0.999) , eps = 1e-08 , weight_decay = 0 , momentum_decay = 0.004 , decoupled_weight_decay = False , * , foreach = None , maximize = False , capturable = False , differentiable = False ) [source] # Optimizer URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.optimizer.Optimizer.html Optimizer class tensorplay.optim.optimizer. Optimizer ( params , defaults ) [source] Base class for optimizers. Parameters : params ( iterable ) – an iterable of Tensor s or dict s. Specifies what Tensors should be optimized. defaults – (dict): a dict containing default values of optimization options (used when a parameter group doesn’t specify them). # tensorplay.optim.optimizer.register_optimizer_step_post_hook URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.optimizer.register_optimizer_step_post_hook.html tensorplay.optim.optimizer.register_optimizer_step_post_hook tensorplay.optim.optimizer. register_optimizer_step_post_hook ( hook : Callable [ [ Optimizer , tuple [ Any , ... ] , dict [ str , Any ] ] , None ] ) → RemovableHandle [source] # tensorplay.optim.optimizer.register_optimizer_step_pre_hook URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.optimizer.register_optimizer_step_pre_hook.html tensorplay.optim.optimizer.register_optimizer_step_pre_hook tensorplay.optim.optimizer. register_optimizer_step_pre_hook ( hook : Callable [ [ Optimizer , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ tuple [ Any , ... ] , dict [ str , Any ] ] | None ] ) → RemovableHandle [source] # RAdam URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.radam.RAdam.html RAdam class tensorplay.optim.radam. RAdam ( params , lr = 0.001 , betas = (0.9, 0.999) , eps = 1e-08 , weight_decay = 0 , decoupled_weight_decay = False , * , foreach = None , maximize = False , capturable = False , differentiable = False ) [source] # RMSprop URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.rmsprop.RMSprop.html RMSprop class tensorplay.optim.rmsprop. RMSprop ( params , lr = 0.01 , alpha = 0.99 , eps = 1e-08 , weight_decay = 0 , momentum = 0 , centered = False , capturable = False , foreach = None , maximize = False , differentiable = False ) [source] RMSprop optimizer matching Torch’s centered and momentum variants. # Rprop URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.rprop.Rprop.html Rprop class tensorplay.optim.rprop. Rprop ( params , lr = 0.01 , etas = (0.5, 1.2) , step_sizes = (1e-06, 50) , * , capturable = False , foreach = None , maximize = False , differentiable = False ) [source] Resilient backpropagation optimizer. # SGD URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.sgd.SGD.html SGD class tensorplay.optim.sgd. SGD ( params : Iterable [ TensorBase ] | Iterable [ dict [ str , Any ] ] | Iterable [ tuple [ str , TensorBase ] ] , lr : float | TensorBase = 0.001 , momentum : float = 0 , dampening : float = 0 , weight_decay : float | TensorBase = 0 , nesterov : bool = False , * , maximize : bool = False , foreach : bool | None = None , differentiable : bool = False , fused : bool | None = None ) [source] # SparseAdam URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.sparse_adam.SparseAdam.html SparseAdam class tensorplay.optim.sparse_adam. SparseAdam ( params : Iterable [ TensorBase ] | Iterable [ dict [ str , Any ] ] | Iterable [ tuple [ str , TensorBase ] ] , lr : float | TensorBase = 0.001 , betas : tuple [ float , float ] = (0.9, 0.999) , eps : float = 1e-08 , maximize : bool = False ) [source] SparseAdam implements a masked version of the Adam algorithm suitable for sparse gradients. Currently, due to implementation constraints (explained below), SparseAdam is only intended for a narrow subset of use cases, specifically parameters of a dense layout with gradients of a sparse layout. This occurs in a special case where the module backwards produces grads already in a sparse layout. One example NN module that behaves as such is nn.Embedding(sparse=True) . SparseAdam approximates the Adam algorithm by masking out the parameter and moment updates corresponding to the zero values in the gradients. Whereas the Adam algorithm will update the first moment, the second moment, and the parameters based on all values of the gradients, SparseAdam only updates the moments and parameters corresponding to the non-zero values of the gradients. A simplified way of thinking about the intended implementation is as such: Create a mask of the non-zero values in the sparse gradients. For example, if your gradient looks like [0, 5, 0, 0, 9], the mask would be [0, 1, 0, 0, 1]. Apply this mask over the running moments and do computation on only the non-zero values. Apply this mask over the parameters and only apply an update on non-zero values. In actuality, we use sparse layout Tensors to optimize this approximation, which means the more gradients that are masked by not being materialized, the more performant the optimization. Since we rely on using sparse layout tensors, we infer that any materialized value in the sparse layout is non-zero and we do NOT actually verify that all values are not zero! It is important to not conflate a semantically sparse tensor (a tensor where many of its values are zeros) with a sparse layout tensor (a tensor where .is_sparse returns True ). The SparseAdam approximation is intended for semantically sparse tensors and the sparse layout is only an implementation detail. A clearer implementation would be to use MaskedTensors, but those are experimental. Note If you suspect your gradients are semantically sparse (but do not have sparse layout), this variant may not be the best for you. Ideally, you want to avoid materializing anything that is suspected to be sparse in the first place, since needing to convert all your grads from dense layout to sparse layout may outweigh the performance gain. Here, using Adam may be the best alternative, unless you can easily rig up your module to output sparse grads similar to nn.Embedding(sparse=True) . If you insist on converting your grads, you can do so by manually overriding your parameters’ .grad fields with their sparse equivalents before calling .step() . Parameters : params ( iterable ) – iterable of parameters or named_parameters to optimize or iterable of dicts defining parameter groups. When using named_parameters, all parameters in all groups should be named lr ( float , Tensor , optional ) – learning rate (default: 1e-3) betas ( Tuple [ float , float ] , optional ) – coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999)) eps ( float , optional ) – term added to the denominator to improve numerical stability (default: 1e-8) maximize ( bool , optional ) – maximize the objective with respect to the params, instead of minimizing (default: False) step ( closure = None ) [source] Perform a single optimization step. Parameters : closure ( Callable , optional ) – A closure that reevaluates the model and returns the loss. # AveragedModel URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.swa_utils.AveragedModel.html AveragedModel class tensorplay.optim.swa_utils. AveragedModel ( model : Module , device : int | tp.device | None = None , avg_fn : Callable [ [ Tensor , Tensor , Tensor | int ] , Tensor ] | None = None , multi_avg_fn : Callable [ [ PARAM_LIST , PARAM_LIST , Tensor | int ] , None ] | None = None , use_buffers = False ) [source] A model that maintains a running SWA or EMA copy of another model. add_module ( name : str , module : Module | None ) → None Add a child module to the current module. The module can be accessed as an attribute using the given name. Parameters : name ( str ) – name of the child module. The child module can be accessed from this module using the given name module ( Module ) – child module to be added to the module. apply ( fn : Callable [ [ Module ] , None ] ) → Self Apply fn recursively to every submodule (as returned by .children() ) as well as self. Typical use includes initializing the parameters of a model (see also nn.init documentation ). Parameters : fn ( Module -> None) – function to be applied to each submodule Returns : self Return type : Module Example: >>> @tensorplay . no_grad () >>> def init_weights ( m ): >>> print ( m ) >>> if type ( m ) == nn . Linear : >>> m . weight . fill_ ( 1.0 ) >>> print ( m . weight ) >>> net = nn . Sequential ( nn . Linear ( 2 , 2 ), nn . Linear ( 2 , 2 )) >>> net . apply ( init_weights ) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Linear(in_features=2, out_features=2, bias=True) Parameter containing: tensor([[1., 1.], [1., 1.]], requires_grad=True) Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) bfloat16 ( ) → Self Casts all floating point parameters and buffers to bfloat16 datatype. Note This method modifies the module in-place. Returns : self Return type : Module buffers ( recurse : bool = True ) → Iterator [ TensorBase ] Return an iterator over module buffers. Parameters : recurse ( bool ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Yields : tensorplay.Tensor – module buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for buf in model . buffers (): >>> print ( type ( buf ), buf . size ()) (20L,) (20L, 1L, 5L, 5L) children ( ) → Iterator [ Module ] Return an iterator over immediate children modules. Yields : Module – a child module compile ( * args , ** kwargs ) Compile this Module’s forward using tensorplay.compile() . This Module’s __call__ method is compiled and all arguments are passed as-is to tensorplay.compile() . See tensorplay.compile() for details on the arguments for this function. cpu ( ) → Self Move all model parameters and buffers to the CPU. Note This method modifies the module in-place. Returns : self Return type : Module cuda ( device : int | Device | None = None ) → Self Move all model parameters and buffers to the GPU. This also makes associated parameters and buffers different objects. So it should be called before constructing the optimizer if the module will live on GPU while being optimized. Note This method modifies the module in-place. Parameters : device ( int , optional ) – if specified, all parameters will be copied to that device Returns : self Return type : Module double ( ) → Self Casts all floating point parameters and buffers to double datatype. Note This method modifies the module in-place. Returns : self Return type : Module eval ( ) → Self Set the module in evaluation mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e. whether they are affected, e.g. Dropout , BatchNorm , etc. This is equivalent with self.train(False) . See Locally disabling gradient computation for a comparison between .eval() and several similar mechanisms that may be confused with it. Returns : self Return type : Module extra_repr ( ) → str Return the extra representation of the module. To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable. float ( ) → Self Casts all floating point parameters and buffers to float datatype. Note This method modifies the module in-place. Returns : self Return type : Module forward ( * args , ** kwargs ) [source] Forward pass. get_buffer ( target : str ) → TensorBase Return the buffer given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the buffer to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The buffer referenced by target Return type : tensorplay.Tensor Raises : AttributeError – If the target string references an invalid path or resolves to something that is not a buffer get_extra_state ( ) → Any Return any extra state to include in the module’s state_dict. Implement this and a corresponding set_extra_state() for your module if you need to store extra state. This function is called when building the module’s state_dict() . Note that extra state should be picklable to ensure working serialization of the state_dict. We only provide backwards compatibility guarantees for serializing Tensors; other objects may break backwards compatibility if their serialized pickled form changes. Returns : Any extra state to store in the module’s state_dict Return type : object get_parameter ( target : str ) → Parameter Return the parameter given by target if it exists, otherwise throw an error. See the docstring for get_submodule for a more detailed explanation of this method’s functionality as well as how to correctly specify target . Parameters : target – The fully-qualified string name of the Parameter to look for. (See get_submodule for how to specify a fully-qualified string.) Returns : The Parameter referenced by target Return type : tensorplay.nn.Parameter Raises : AttributeError – If the target string references an invalid path or resolves to something that is not an nn.Parameter get_submodule ( target : str ) → Module Return the submodule given by target if it exists, otherwise throw an error. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(16, 33, kernel_size=(3, 3), stride=(2, 2)) ) (linear): Linear(in_features=100, out_features=200, bias=True) ) ) (The diagram shows an nn.Module A . A which has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To check whether or not we have the linear submodule, we would call get_submodule("net_b.linear") . To check whether we have the conv submodule, we would call get_submodule("net_b.net_c.conv") . The runtime of get_submodule is bounded by the degree of module nesting in target . A query against named_modules achieves the same result, but it is O(N) in the number of transitive modules. So, for a simple check to see if some submodule exists, get_submodule should always be used. Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) Returns : The submodule referenced by target Return type : tensorplay.nn.Module Raises : AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . half ( ) → Self Casts all floating point parameters and buffers to half datatype. Note This method modifies the module in-place. Returns : self Return type : Module load_state_dict ( state_dict : Mapping [ str , Any ] , strict : bool = True , assign : bool = False ) Copy parameters and buffers from state_dict into this module and its descendants. If strict is True , then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function. Warning If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True . Parameters : state_dict ( dict ) – a dict containing parameters and persistent buffers. strict ( bool , optional ) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True assign ( bool , optional ) – When set to False , the properties of the tensors in the current module are preserved whereas setting it to True preserves properties of the Tensors in the state dict. The only exception is the requires_grad field of Parameter for which the value from the module is preserved. Default: False Returns : missing_keys is a list of str containing any keys that are expected by this module but missing from the provided state_dict . unexpected_keys is a list of str containing the keys that are not expected by this module but present in the provided state_dict . Return type : NamedTuple with missing_keys and unexpected_keys fields Note If a parameter or buffer is registered as None and its corresponding key exists in state_dict , load_state_dict() will raise a RuntimeError . modules ( ) → Iterator [ Module ] Return an iterator over all modules in the network. Yields : Module – a module in the network Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . modules ()): ... print ( idx , '->' , m ) 0 -> Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) ) 1 -> Linear(in_features=2, out_features=2, bias=True) named_buffers ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , TensorBase ] ] Return an iterator over module buffers, yielding both the name of the buffer as well as the buffer itself. Parameters : prefix ( str ) – prefix to prepend to all buffer names. recurse ( bool , optional ) – if True, then yields buffers of this module and all submodules. Otherwise, yields only buffers that are direct members of this module. Defaults to True. remove_duplicate ( bool , optional ) – whether to remove the duplicated buffers in the result. Defaults to True. Yields : (str, tensorplay.Tensor) – Tuple containing the name and buffer Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , buf in self . named_buffers (): >>> if name in [ 'running_var' ]: >>> print ( buf . size ()) named_children ( ) → Iterator [ tuple [ str , Module ] ] Return an iterator over immediate children modules, yielding both the name of the module as well as the module itself. Yields : (str, Module) – Tuple containing a name and child module Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , module in model . named_children (): >>> if name in [ 'conv4' , 'conv5' ]: >>> print ( module ) named_modules ( memo : set [ Module ] | None = None , prefix : str = '' , remove_duplicate : bool = True ) Return an iterator over all modules in the network, yielding both the name of the module as well as the module itself. Parameters : memo – a memo to store the set of modules already added to the result prefix – a prefix that will be added to the name of the module remove_duplicate – whether to remove the duplicated module instances in the result or not Yields : (str, Module) – Tuple of name and module Note Duplicate modules are returned only once. In the following example, l will be returned only once. Example: >>> l = nn . Linear ( 2 , 2 ) >>> net = nn . Sequential ( l , l ) >>> for idx , m in enumerate ( net . named_modules ()): ... print ( idx , '->' , m ) 0 -> ('', Sequential( (0): Linear(in_features=2, out_features=2, bias=True) (1): Linear(in_features=2, out_features=2, bias=True) )) 1 -> ('0', Linear(in_features=2, out_features=2, bias=True)) named_parameters ( prefix : str = '' , recurse : bool = True , remove_duplicate : bool = True ) → Iterator [ tuple [ str , Parameter ] ] Return an iterator over module parameters, yielding both the name of the parameter as well as the parameter itself. Parameters : prefix ( str ) – prefix to prepend to all parameter names. recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. remove_duplicate ( bool , optional ) – whether to remove the duplicated parameters in the result. Defaults to True. Yields : (str, Parameter) – Tuple containing the name and parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for name , param in self . named_parameters (): >>> if name in [ 'bias' ]: >>> print ( param . size ()) parameters ( recurse : bool = True ) → Iterator [ Parameter ] Return an iterator over module parameters. This is typically passed to an optimizer. Parameters : recurse ( bool ) – if True, then yields parameters of this module and all submodules. Otherwise, yields only parameters that are direct members of this module. Yields : Parameter – module parameter Example: >>> # xdoctest: +SKIP("undefined vars") >>> for param in model . parameters (): >>> print ( type ( param ), param . size ()) (20L,) (20L, 1L, 5L, 5L) register_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] ) → RemovableHandle Register a backward hook on the module. This function is deprecated in favor of register_full_backward_hook() and the behavior of this function will change in future versions. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_buffer ( name : str , tensor : TensorBase | None , persistent : bool = True ) → None Add a buffer to the module. This is typically used to register a buffer that should not be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the module’s state. Buffers, by default, are persistent and will be saved alongside parameters. This behavior can be changed by setting persistent to False . The only difference between a persistent buffer and a non-persistent buffer is that the latter will not be a part of this module’s state_dict . Buffers can be accessed as attributes using given names. Parameters : name ( str ) – name of the buffer. The buffer can be accessed from this module using the given name tensor ( Tensor or None ) – buffer to be registered. If None , then operations that run on buffers, such as cuda , are ignored. If None , the buffer is not included in the module’s state_dict . persistent ( bool ) – whether the buffer is part of this module’s state_dict . Example: >>> # xdoctest: +SKIP("undefined vars") >>> self . register_buffer ( 'running_mean' , tensorplay . zeros ( num_features )) register_forward_hook ( hook : Callable [ [ T , tuple [ Any , ... ] , Any ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] , Any ] , Any | None ] , * , prepend : bool = False , with_kwargs : bool = False , always_call : bool = False ) → RemovableHandle Register a forward hook on the module. The hook will be called every time after forward() has computed an output. If with_kwargs is False or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the output. It can modify the input inplace but it will not have effect on forward since this is called after forward() is called. The hook should have the following signature: hook ( module , args , output ) -> None or modified output If with_kwargs is True , the forward hook will be passed the kwargs given to the forward function and be expected to return the output possibly modified. The hook should have the following signature: hook ( module , args , kwargs , output ) -> None or modified output Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If True , the provided hook will be fired before all existing forward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward hooks on this tensorplay.nn.Module . Note that global forward hooks registered with register_module_forward_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If True , the hook will be passed the kwargs given to the forward function. Default: False always_call ( bool ) – If True the hook will be run regardless of whether an exception is raised while calling the Module. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_forward_pre_hook ( hook : Callable [ [ T , tuple [ Any , ... ] ] , Any | None ] | Callable [ [ T , tuple [ Any , ... ] , dict [ str , Any ] ] , tuple [ Any , dict [ str , Any ] ] | None ] , * , prepend : bool = False , with_kwargs : bool = False ) → RemovableHandle Register a forward pre-hook on the module. The hook will be called every time before forward() is invoked. If with_kwargs is false or not specified, the input contains only the positional arguments given to the module. Keyword arguments won’t be passed to the hooks and only to the forward . The hook can modify the input. User can either return a tuple or a single modified value in the hook. We will wrap the value into a tuple if a single value is returned (unless that value is already a tuple). The hook should have the following signature: hook ( module , args ) -> None or modified input If with_kwargs is true, the forward pre-hook will be passed the kwargs given to the forward function. And if the hook modifies the input, both the args and kwargs should be returned. The hook should have the following signature: hook ( module , args , kwargs ) -> None or a tuple of modified input and kwargs Parameters : hook ( Callable ) – The user defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing forward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing forward_pre hooks on this tensorplay.nn.Module . Note that global forward_pre hooks registered with register_module_forward_pre_hook() will fire before all hooks registered by this method. Default: False with_kwargs ( bool ) – If true, the hook will be passed the kwargs given to the forward function. Default: False Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward hook on the module. The hook will be called every time the gradients with respect to a module are computed, and its firing rules are as follows: Ordinarily, the hook fires when the gradients are computed with respect to the module inputs. If none of the module inputs require gradients, the hook will fire when the gradients are computed with respect to module outputs. If none of the module outputs require gradients, then the hooks will not fire. The hook should have the following signature: hook ( module , grad_input , grad_output ) -> tuple ( Tensor ) or None The grad_input and grad_output are tuples that contain the gradients with respect to the inputs and outputs respectively. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations. grad_input will only correspond to the inputs given as positional arguments and all kwarg arguments are ignored. Entries in grad_input and grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs or outputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward hooks on this tensorplay.nn.Module . Note that global backward hooks registered with register_module_full_backward_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_full_backward_pre_hook ( hook : Callable [ [ Module , tuple [ TensorBase , ... ] | TensorBase ] , None | tuple [ TensorBase , ... ] | TensorBase ] , prepend : bool = False ) → RemovableHandle Register a backward pre-hook on the module. The hook will be called every time the gradients for the module are computed. The hook should have the following signature: hook ( module , grad_output ) -> tuple [ Tensor ] or None The grad_output is a tuple. The hook should not modify its arguments, but it can optionally return a new gradient with respect to the output that will be used in place of grad_output in subsequent computations. Entries in grad_output will be None for all non-Tensor arguments. For technical reasons, when this hook is applied to a Module, its forward function will receive a view of each Tensor passed to the Module. Similarly the caller will receive a view of each Tensor returned by the Module’s forward function. Warning Modifying inputs inplace is not allowed when using backward hooks and will raise an error. Parameters : hook ( Callable ) – The user-defined hook to be registered. prepend ( bool ) – If true, the provided hook will be fired before all existing backward_pre hooks on this tensorplay.nn.Module . Otherwise, the provided hook will be fired after all existing backward_pre hooks on this tensorplay.nn.Module . Note that global backward_pre hooks registered with register_module_full_backward_pre_hook() will fire before all hooks registered by this method. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_post_hook ( hook ) Register a post-hook to be run after module’s load_state_dict() is called. It should have the following signature:: hook(module, incompatible_keys) -> None The module argument is the current module that this hook is registered on, and the incompatible_keys argument is a NamedTuple consisting of attributes missing_keys and unexpected_keys . missing_keys is a list of str containing the missing keys and unexpected_keys is a list of str containing the unexpected keys. The given incompatible_keys can be modified inplace if needed. Note that the checks performed when calling load_state_dict() with strict=True are affected by modifications the hook makes to missing_keys or unexpected_keys , as expected. Additions to either set of keys will result in an error being thrown when strict=True , and clearing out both missing and unexpected keys will avoid an error. Returns : a handle that can be used to remove the added hook by calling handle.remove() Return type : tensorplay.utils.hooks.RemovableHandle register_load_state_dict_pre_hook ( hook ) Register a pre-hook to be run before module’s load_state_dict() is called. It should have the following signature:: hook(module, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs) -> None # noqa: B950 Parameters : hook ( Callable ) – Callable hook that will be invoked before loading the state dict. register_module ( name : str , module : Module | None ) → None Alias for add_module() . register_parameter ( name : str , param : Parameter | None ) → None Add a parameter to the module. The parameter can be accessed as an attribute using given name. Parameters : name ( str ) – name of the parameter. The parameter can be accessed from this module using the given name param ( Parameter or None ) – parameter to be added to the module. If None , then operations that run on parameters, such as cuda , are ignored. If None , the parameter is not included in the module’s state_dict . register_state_dict_post_hook ( hook ) Register a post-hook for the state_dict() method. It should have the following signature:: hook(module, state_dict, prefix, local_metadata) -> None The registered hooks can modify the state_dict inplace. register_state_dict_pre_hook ( hook ) Register a pre-hook for the state_dict() method. It should have the following signature:: hook(module, prefix, keep_vars) -> None The registered hooks can be used to perform pre-processing before the state_dict call is made. requires_grad_ ( requires_grad : bool = True ) → Self Change if autograd should record operations on parameters in this module. This method sets the parameters’ requires_grad attributes in-place. This method is helpful for freezing part of the module for finetuning or training parts of a model individually (e.g., GAN training). See Locally disabling gradient computation for a comparison between .requires_grad_() and several similar mechanisms that may be confused with it. Parameters : requires_grad ( bool ) – whether autograd should record operations on parameters in this module. Default: True . Returns : self Return type : Module set_extra_state ( state : Any ) → None Set extra state contained in the loaded state_dict . This function is called from load_state_dict() to handle any extra state found within the state_dict . Implement this function and a corresponding get_extra_state() for your module if you need to store extra state within its state_dict . Parameters : state ( dict ) – Extra state from the state_dict set_submodule ( target : str , module : Module , strict : bool = False ) → None Set the submodule given by target if it exists, otherwise throw an error. Note If strict is set to False (default), the method will replace an existing submodule or create a new submodule if the parent module exists. If strict is set to True , the method will only attempt to replace an existing submodule and throw an error if the submodule does not exist. For example, let’s say you have an nn.Module A that looks like this: A( (net_b): Module( (net_c): Module( (conv): Conv2d(3, 3, 3) ) (linear): Linear(3, 3) ) ) (The diagram shows an nn.Module A . A has a nested submodule net_b , which itself has two submodules net_c and linear . net_c then has a submodule conv .) To override the Conv2d with a new submodule Linear , you could call set_submodule("net_b.net_c.conv", nn.Linear(1, 1)) where strict could be True or False To add a new submodule Conv2d to the existing net_b module, you would call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1)) . In the above if you set strict=True and call set_submodule("net_b.conv", nn.Conv2d(1, 1, 1), strict=True) , an AttributeError will be raised because net_b does not have a submodule named conv . Parameters : target – The fully-qualified string name of the submodule to look for. (See above example for how to specify a fully-qualified string.) module – The module to set the submodule to. strict – If False , the method will replace an existing submodule or create a new submodule if the parent module exists. If True , the method will only attempt to replace an existing submodule and throw an error if the submodule doesn’t already exist. Raises : ValueError – If the target string is empty or if module is not an instance of nn.Module . AttributeError – If at any point along the path resulting from the target string the (sub)path resolves to a non-existent attribute name or an object that is not an instance of nn.Module . share_memory ( ) → Self See tensorplay.Tensor.share_memory_() . state_dict ( * args , destination = None , prefix = '' , keep_vars = False ) Return a dictionary containing references to the whole state of the module. Both parameters and persistent buffers (e.g. running averages) are included. Keys are corresponding parameter and buffer names. Parameters and buffers set to None are not included. Note The returned object is a shallow copy. It contains references to the module’s parameters and buffers. Warning Currently state_dict() also accepts positional arguments for destination , prefix and keep_vars in order. However, this is being deprecated and keyword arguments will be enforced in future releases. Warning Please avoid the use of argument destination as it is not designed for end-users. Parameters : destination ( dict , optional ) – If provided, the state of module will be updated into the dict and the same object is returned. Otherwise, an OrderedDict will be created and returned. Default: None . prefix ( str , optional ) – a prefix added to parameter and buffer names to compose the keys in state_dict. Default: '' . keep_vars ( bool , optional ) – by default the Tensor s returned in the state dict are detached from autograd. If it’s set to True , detaching will not be performed. Default: False . Returns : a dictionary containing a whole state of the module Return type : dict Example: >>> # xdoctest: +SKIP("undefined vars") >>> module . state_dict () . keys () ['bias', 'weight'] to ( * args , ** kwargs ) Move and/or cast the parameters and buffers. This can be called as to ( device = None , dtype = None , non_blocking = False ) to ( dtype , non_blocking = False ) to ( tensor , non_blocking = False ) to ( memory_format = tensorplay.channels_last ) Its signature is similar to tensorplay.Tensor.to() , but only accepts floating point or complex dtype s. In addition, this method will only cast the floating point or complex parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device , if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. See below for examples. Note This method modifies the module in-place. Parameters : device ( tensorplay.device ) – the desired device of the parameters and buffers in this module dtype ( tensorplay.dtype ) – the desired floating point or complex dtype of the parameters and buffers in this module tensor ( tensorplay.Tensor ) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module memory_format ( tensorplay.memory_format ) – the desired memory format for 4D parameters and buffers in this module (keyword only argument) Returns : self Return type : Module Examples: >>> # xdoctest: +IGNORE_WANT("non-deterministic") >>> linear = nn . Linear ( 2 , 2 ) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]]) >>> linear . to ( tensorplay . double ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1913, -0.3420], [-0.5113, -0.2325]], dtype=tensorplay.float64) >>> # xdoctest: +REQUIRES(env:TENSORPLAY_DOCTEST_CUDA1) >>> gpu1 = tensorplay . device ( "cuda:1" ) >>> linear . to ( gpu1 , dtype = tensorplay . half , non_blocking = True ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16, device='cuda:1') >>> cpu = tensorplay . device ( "cpu" ) >>> linear . to ( cpu ) Linear(in_features=2, out_features=2, bias=True) >>> linear . weight Parameter containing: tensor([[ 0.1914, -0.3420], [-0.5112, -0.2324]], dtype=tensorplay.float16) >>> linear = nn . Linear ( 2 , 2 , bias = None ) . to ( tensorplay . cdouble ) >>> linear . weight Parameter containing: tensor([[ 0.3741+0.j, 0.2382+0.j], [ 0.5593+0.j, -0.4443+0.j]], dtype=tensorplay.complex128) >>> linear ( tensorplay . ones ( 3 , 2 , dtype = tensorplay . cdouble )) tensor([[0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j], [0.6122+0.j, 0.1150+0.j]], dtype=tensorplay.complex128) to_empty ( * , device : str | Device | int | None , recurse : bool = True ) → Self Move the parameters and buffers to the specified device without copying storage. Parameters : device ( tensorplay.device ) – The desired device of the parameters and buffers in this module. recurse ( bool ) – Whether parameters and buffers of submodules should be recursively moved to the specified device. Returns : self Return type : Module train ( mode : bool = True ) → Self Set the module in training mode. This has an effect only on certain modules. See the documentation of particular modules for details of their behaviors in training/evaluation mode, i.e., whether they are affected, e.g. Dropout , BatchNorm , etc. Parameters : mode ( bool ) – whether to set training mode ( True ) or evaluation mode ( False ). Default: True . Returns : self Return type : Module type ( dst_type : DType | str ) → Self Casts all parameters and buffers to dst_type . Note This method modifies the module in-place. Parameters : dst_type ( type or string ) – the desired type Returns : self Return type : Module update_parameters ( model : Module ) → None [source] Update model parameters. zero_grad ( set_to_none : bool = True ) → None Reset gradients of all model parameters. See similar function under tensorplay.optim.Optimizer for more context. Parameters : set_to_none ( bool ) – instead of setting to zero, set the grads to None. See tensorplay.optim.Optimizer.zero_grad() for details. # tensorplay.optim.swa_utils.get_ema_avg_fn URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.swa_utils.get_ema_avg_fn.html tensorplay.optim.swa_utils.get_ema_avg_fn tensorplay.optim.swa_utils. get_ema_avg_fn ( decay = 0.999 ) [source] # tensorplay.optim.swa_utils.get_ema_multi_avg_fn URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.swa_utils.get_ema_multi_avg_fn.html tensorplay.optim.swa_utils.get_ema_multi_avg_fn tensorplay.optim.swa_utils. get_ema_multi_avg_fn ( decay = 0.999 ) [source] # tensorplay.optim.swa_utils.get_swa_avg_fn URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.swa_utils.get_swa_avg_fn.html tensorplay.optim.swa_utils.get_swa_avg_fn tensorplay.optim.swa_utils. get_swa_avg_fn ( ) [source] # tensorplay.optim.swa_utils.get_swa_multi_avg_fn URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.swa_utils.get_swa_multi_avg_fn.html tensorplay.optim.swa_utils.get_swa_multi_avg_fn tensorplay.optim.swa_utils. get_swa_multi_avg_fn ( ) [source] # SWALR URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.swa_utils.SWALR.html SWALR class tensorplay.optim.swa_utils. SWALR ( optimizer : Optimizer , swa_lr : float , anneal_epochs = 10 , anneal_strategy : Literal [ 'cos' , 'linear' ] = 'cos' , last_epoch = -1 ) [source] Anneal each optimizer learning rate to a fixed SWA learning rate. get_last_lr ( ) → list [ float | TensorBase ] Get the most recent learning rates computed by this scheduler. Returns : A list of learning rates with entries for each of the optimizer’s param_groups , with the same types as their group["lr"] s. Return type : list [ float | Tensor] Note The returned Tensor s are copies, and never alias the optimizer’s group["lr"] s. step ( epoch : int | None = None ) → None Step the scheduler. Parameters : epoch ( int , optional ) – Deprecated since version 1.4: If provided, sets last_epoch to epoch and uses _get_closed_form_lr() if it is available. This is not universally supported. Use step() without arguments instead. Note Call this method after calling the optimizer’s step() . # tensorplay.optim.swa_utils.update_bn URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.swa_utils.update_bn.html tensorplay.optim.swa_utils.update_bn tensorplay.optim.swa_utils. update_bn ( loader : Iterable [ Any ] , model : Module , device : int | tp.device | None = None ) → None [source] # tensorplay.optim.swap_in_optimizer_params_and_state URL: https://www.tensorplay.cn/docs/generated/tensorplay.optim.swap_in_optimizer_params_and_state.html tensorplay.optim.swap_in_optimizer_params_and_state tensorplay.optim. swap_in_optimizer_params_and_state ( optimizer , swapin_parameters : dict [ str , Any ] , swapin_optim_state : dict [ str , Any ] ) Temporarily install replacement parameters and packed optimizer state. # tensorplay.pixel_shuffle URL: https://www.tensorplay.cn/docs/generated/tensorplay.pixel_shuffle.html tensorplay.pixel_shuffle tensorplay. pixel_shuffle ( input , upscale_factor ) [source] # tensorplay.quantized_linear URL: https://www.tensorplay.cn/docs/generated/tensorplay.quantized_linear.html tensorplay.quantized_linear tensorplay. quantized_linear ( input , weight , input_scale , input_zero_point , weight_scales , weight_zero_points , bias = None ) [source] # tensorplay.random_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.random_.html tensorplay.random_ tensorplay. random_ ( input , low = 0 , high = 0 ) [source] # tensorplay.random.fork_rng URL: https://www.tensorplay.cn/docs/generated/tensorplay.random.fork_rng.html tensorplay.random.fork_rng tensorplay.random. fork_rng ( devices = None , enabled = True , _caller = 'fork_rng' , _devices_kw = 'devices' ) [source] Forks the RNG state: code inside the context gets a pristine RNG. Saves the CPU RNG state on entry and restores it on exit, so random operations inside the block do not advance the outer stream (mirrors torch.random.fork_rng ). CUDA devices are not forked; pass devices=[] explicitly to acknowledge that (same contract as torch). # tensorplay.reflection_pad_nd URL: https://www.tensorplay.cn/docs/generated/tensorplay.reflection_pad_nd.html tensorplay.reflection_pad_nd tensorplay. reflection_pad_nd ( input , pad ) [source] # tensorplay.relu URL: https://www.tensorplay.cn/docs/generated/tensorplay.relu.html tensorplay.relu tensorplay. relu ( input ) [source] # tensorplay.relu6 URL: https://www.tensorplay.cn/docs/generated/tensorplay.relu6.html tensorplay.relu6 tensorplay. relu6 ( input ) [source] # tensorplay.repeat URL: https://www.tensorplay.cn/docs/generated/tensorplay.repeat.html tensorplay.repeat tensorplay. repeat ( input , repeats ) [source] # tensorplay.replication_pad_nd URL: https://www.tensorplay.cn/docs/generated/tensorplay.replication_pad_nd.html tensorplay.replication_pad_nd tensorplay. replication_pad_nd ( input , pad ) [source] # tensorplay.resize_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.resize_.html tensorplay.resize_ tensorplay. resize_ ( input , size ) [source] # tensorplay.sample URL: https://www.tensorplay.cn/docs/generated/tensorplay.sample.html tensorplay.sample tensorplay. sample ( logits , temperature = 1.0 , top_k = 0 , top_p = 1.0 , impl = 0 ) [source] # Scalar URL: https://www.tensorplay.cn/docs/generated/tensorplay.Scalar.html Scalar class tensorplay. Scalar # tensorplay.scaled_dot_product_attention URL: https://www.tensorplay.cn/docs/generated/tensorplay.scaled_dot_product_attention.html tensorplay.scaled_dot_product_attention tensorplay. scaled_dot_product_attention ( query , key , value , is_causal = False , impl = 0 ) [source] # tensorplay.scaled_modified_bessel_k0 URL: https://www.tensorplay.cn/docs/generated/tensorplay.scaled_modified_bessel_k0.html tensorplay.scaled_modified_bessel_k0 tensorplay. scaled_modified_bessel_k0 ( input ) [source] # tensorplay.scaled_modified_bessel_k1 URL: https://www.tensorplay.cn/docs/generated/tensorplay.scaled_modified_bessel_k1.html tensorplay.scaled_modified_bessel_k1 tensorplay. scaled_modified_bessel_k1 ( input ) [source] # tensorplay.scatter_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.scatter_.html tensorplay.scatter_ tensorplay. scatter_ ( input , dim , index , src ) [source] # tensorplay.scatter_add_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.scatter_add_.html tensorplay.scatter_add_ tensorplay. scatter_add_ ( input , dim , index , src ) [source] # tensorplay.seed URL: https://www.tensorplay.cn/docs/generated/tensorplay.seed.html tensorplay.seed tensorplay. seed ( ) → int # tensorplay.serialization.load URL: https://www.tensorplay.cn/docs/generated/tensorplay.serialization.load.html tensorplay.serialization.load tensorplay.serialization. load ( f : str | PathLike [ str ] , map_location : Any | None = None , * , mmap : bool = False , weights_only : bool = True ) → Any [source] Load a checkpoint saved by tp.save or torch.save . Format is detected by extension or content sniffing: .mega (and MEGA shard indexes), .safetensors , and torch archives ( .pt / .pth – zip, magic-number stream, and tar layouts). Parameters : f – path or file-like object (seekable binary). map_location – controls where tensors land. Accepts a device string / tensorplay.Device (all tensors), a mapping of saved location -> target, or a callable invoked per storage as fn(storage_stub, location) returning None , a device, or a TensorPlay tensor receiving the data. mmap – if true, MEGA and safetensors tensors are zero-copy views over a private (copy-on-write) mapping of the file; the returned containers keep the mapping alive, and bytes are paged in on first touch. Explicit device remapping still copies. Torch archives load eagerly. weights_only – accepted for API familiarity; every TensorPlay loader is weights-only by construction. Passing False warns. # tensorplay.serialization.save URL: https://www.tensorplay.cn/docs/generated/tensorplay.serialization.save.html tensorplay.serialization.save tensorplay.serialization. save ( obj : Any , f : str | PathLike [ str ] , * , metadata : Mapping [ str , Any ] | None = None , alignment : int = 4096 , checksum : str = 'crc32' , pickle_protocol : int = 2 ) → None [source] Save obj to disk. Formats are selected by extension: .mega (default): native MEGA artifact. Supports nested tensor containers plus JSON-primitive leaves, shared-storage deduplication, an alignment , per-region checksum ( "crc32" , "sha256" , or "none" ) and free-form metadata (strings, numbers, bools, lists). .safetensors : flat name->tensor mapping. .pt / .pth : torch-compatible archive; any picklable object graph. A file-like object ( io.BytesIO etc.) receives the torch pickle format, matching torch.save conventions. # tensorplay.set_autocast_cache_enabled URL: https://www.tensorplay.cn/docs/generated/tensorplay.set_autocast_cache_enabled.html tensorplay.set_autocast_cache_enabled tensorplay. set_autocast_cache_enabled ( enabled : bool ) → None # tensorplay.set_autocast_dtype URL: https://www.tensorplay.cn/docs/generated/tensorplay.set_autocast_dtype.html tensorplay.set_autocast_dtype tensorplay. set_autocast_dtype ( device_type : str , dtype : tensorplay._C.DType ) → None # tensorplay.set_autocast_enabled URL: https://www.tensorplay.cn/docs/generated/tensorplay.set_autocast_enabled.html tensorplay.set_autocast_enabled tensorplay. set_autocast_enabled ( device_type : str , enabled : bool ) → None # tensorplay.set_default_device URL: https://www.tensorplay.cn/docs/generated/tensorplay.set_default_device.html tensorplay.set_default_device tensorplay. set_default_device ( device : Device ) → None [source] Sets the default tensorplay.Tensor to be allocated on device . This does not affect factory function calls which are called with an explicit device argument. Factory calls will be performed as if they were passed device as an argument. To only temporarily change the default device instead of setting it globally, use with tensorplay.device(device): instead. The default device is initially cpu . If you set the default tensor device to another device (e.g., cuda ) without a device index, tensors will be allocated on whatever the current device for the device type, even after tensorplay.cuda.set_device() is called. Warning This function imposes a slight performance cost on every Python call to the tensorplay API (not just factory functions). Note This doesn’t affect functions that create tensors that share the same memory as the input, like: tensorplay.from_numpy() and tensorplay.frombuffer() Parameters : device ( tensorplay.device , str, int, or None) – the device to set as default, or None to clear the override. An integer is interpreted as an index for the current accelerator. Example: >>> # xdoctest: +SKIP("requires cuda, changes global state") >>> tensorplay . get_default_device () device(type='cpu') >>> tensorplay . set_default_device ( 'cuda' ) # current device is 0 >>> tensorplay . get_default_device () device(type='cuda', index=0) >>> tensorplay . set_default_device ( 'cuda' ) >>> tensorplay . cuda . set_device ( 'cuda:1' ) # current device is 1 >>> tensorplay . get_default_device () device(type='cuda', index=1) >>> tensorplay . set_default_device ( 'cuda:1' ) >>> tensorplay . get_default_device () device(type='cuda', index=1) # tensorplay.set_default_dtype URL: https://www.tensorplay.cn/docs/generated/tensorplay.set_default_dtype.html tensorplay.set_default_dtype tensorplay. set_default_dtype ( d : DType , / ) → None [source] Sets the default floating point dtype to d . Supports floating point dtype as inputs. Other dtypes will cause tensorplay to raise an exception. When TensorPlay is initialized its default floating point dtype is float32, and the intent of set_default_dtype(float64) is to facilitate NumPy-like type inference. The default floating point dtype is used to: Implicitly determine the default complex dtype. When the default floating type is float16, the default complex dtype is complex32. For float32, the default complex dtype is complex64. For float64, it is complex128. For bfloat16, an exception will be raised because there is no corresponding complex type for bfloat16. Infer the dtype for tensors constructed using Python floats or complex Python numbers. See examples below. Determine the result of type promotion between bool and integer tensors and Python floats and complex Python numbers. Parameters : d ( tensorplay.dtype ) – the floating point dtype to make the default. Example >>> # xdoctest: +SKIP("Other tests may have changed the default type. Can we reset it?") >>> # initial default for floating point is float32 >>> # Python floats are interpreted as float32 >>> tensorplay . tensor ([ 1.2 , 3 ]) . dtype tensorplay.float32 >>> # initial default for floating point is complex64 >>> # Complex Python numbers are interpreted as complex64 >>> tensorplay . tensor ([ 1.2 , 3 j ]) . dtype tensorplay.complex64 >>> tensorplay . set_default_dtype ( tensorplay . float64 ) >>> # Python floats are now interpreted as float64 >>> tensorplay . tensor ([ 1.2 , 3 ]) . dtype # a new floating point tensor tensorplay.float64 >>> # Complex Python numbers are now interpreted as complex128 >>> tensorplay . tensor ([ 1.2 , 3 j ]) . dtype # a new complex tensor tensorplay.complex128 >>> tensorplay . set_default_dtype ( tensorplay . float16 ) >>> # Python floats are now interpreted as float16 >>> tensorplay . tensor ([ 1.2 , 3 ]) . dtype # a new floating point tensor tensorplay.float16 >>> # Complex Python numbers are now interpreted as complex128 >>> tensorplay . tensor ([ 1.2 , 3 j ]) . dtype # a new complex tensor tensorplay.complex32 # tensorplay.set_deterministic_debug_mode URL: https://www.tensorplay.cn/docs/generated/tensorplay.set_deterministic_debug_mode.html tensorplay.set_deterministic_debug_mode tensorplay. set_deterministic_debug_mode ( debug_mode : int | str ) → None [source] Sets the debug mode for deterministic operations. Note This is an alternative interface for tensorplay.use_deterministic_algorithms() . Refer to that function’s documentation for details about affected operations. Parameters : debug_mode ( str or int ) – If “default” or 0, don’t error or warn on nondeterministic operations. If “warn” or 1, warn on nondeterministic operations. If “error” or 2, error on nondeterministic operations. # tensorplay.set_float32_matmul_precision URL: https://www.tensorplay.cn/docs/generated/tensorplay.set_float32_matmul_precision.html tensorplay.set_float32_matmul_precision tensorplay. set_float32_matmul_precision ( precision : str ) → None [source] Sets the internal precision of float32 matrix multiplications. Running float32 matrix multiplications in lower precision may significantly increase performance, and in some programs the loss of precision has a negligible impact. Supports three settings: “highest”, float32 matrix multiplications use the float32 datatype (24 mantissa bits with 23 bits explicitly stored) for internal computations. “high”, float32 matrix multiplications either use the TensorFloat32 datatype (10 mantissa bits explicitly stored) or treat each float32 number as the sum of two bfloat16 numbers (approximately 16 mantissa bits with 14 bits explicitly stored), if the appropriate fast matrix multiplication algorithms are available. Otherwise float32 matrix multiplications are computed as if the precision is “highest”. See below for more information on the bfloat16 approach. “medium”, float32 matrix multiplications use the bfloat16 datatype (8 mantissa bits with 7 bits explicitly stored) for internal computations, if a fast matrix multiplication algorithm using that datatype internally is available. Otherwise float32 matrix multiplications are computed as if the precision is “high”. Note This does not change the output dtype of float32 matrix multiplications, it controls how the internal computation of the matrix multiplication is performed. Note This does not change the precision of convolution operations. Other flags, like tensorplay.backends.cudnn.allow_tf32 , may control the precision of convolution operations. Note This flag currently only affects one native device type: CUDA. If “high” or “medium” are set then the TensorFloat32 datatype will be used when computing float32 matrix multiplications, equivalent to setting tensorplay.backends.cuda.matmul.allow_tf32 = True . When “highest” (the default) is set then the float32 datatype is used for internal computations, equivalent to setting tensorplay.backends.cuda.matmul.allow_tf32 = False . Parameters : precision ( str ) – can be set to “highest” (default), “high”, or “medium” (see above). # tensorplay.set_num_threads URL: https://www.tensorplay.cn/docs/generated/tensorplay.set_num_threads.html tensorplay.set_num_threads tensorplay. set_num_threads ( nthreads : SupportsInt | SupportsIndex ) → None Sets the number of threads used for intraop parallelism on CPU. # tensorplay.set_printoptions URL: https://www.tensorplay.cn/docs/generated/tensorplay.set_printoptions.html tensorplay.set_printoptions tensorplay. set_printoptions ( edge_items : SupportsInt | SupportsIndex = -1 , threshold : SupportsInt | SupportsIndex = -1 , precision : SupportsInt | SupportsIndex = -1 , linewidth : SupportsInt | SupportsIndex = -1 ) → None Set print options # tensorplay.set_rng_state URL: https://www.tensorplay.cn/docs/generated/tensorplay.set_rng_state.html tensorplay.set_rng_state tensorplay. set_rng_state ( new_state: tensorplay::Tensor ) → None # tensorplay.shifted_chebyshev_polynomial_t URL: https://www.tensorplay.cn/docs/generated/tensorplay.shifted_chebyshev_polynomial_t.html tensorplay.shifted_chebyshev_polynomial_t tensorplay. shifted_chebyshev_polynomial_t ( x , n ) [source] # tensorplay.shifted_chebyshev_polynomial_u URL: https://www.tensorplay.cn/docs/generated/tensorplay.shifted_chebyshev_polynomial_u.html tensorplay.shifted_chebyshev_polynomial_u tensorplay. shifted_chebyshev_polynomial_u ( x , n ) [source] # tensorplay.shifted_chebyshev_polynomial_v URL: https://www.tensorplay.cn/docs/generated/tensorplay.shifted_chebyshev_polynomial_v.html tensorplay.shifted_chebyshev_polynomial_v tensorplay. shifted_chebyshev_polynomial_v ( x , n ) [source] # tensorplay.shifted_chebyshev_polynomial_w URL: https://www.tensorplay.cn/docs/generated/tensorplay.shifted_chebyshev_polynomial_w.html tensorplay.shifted_chebyshev_polynomial_w tensorplay. shifted_chebyshev_polynomial_w ( x , n ) [source] # tensorplay.silu URL: https://www.tensorplay.cn/docs/generated/tensorplay.silu.html tensorplay.silu tensorplay. silu ( input ) [source] # Size URL: https://www.tensorplay.cn/docs/generated/tensorplay.Size.html Size class tensorplay. Size # tensorplay.slice URL: https://www.tensorplay.cn/docs/generated/tensorplay.slice.html tensorplay.slice tensorplay. slice ( input , dim = 0 , start = None , end = None , step = 1 ) [source] # tensorplay.smooth_l1_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.smooth_l1_loss.html tensorplay.smooth_l1_loss tensorplay. smooth_l1_loss ( input , target , beta = 1.0 ) [source] # tensorplay.soft_margin_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.soft_margin_loss.html tensorplay.soft_margin_loss tensorplay. soft_margin_loss ( input , target ) [source] # tensorplay.softplus URL: https://www.tensorplay.cn/docs/generated/tensorplay.softplus.html tensorplay.softplus tensorplay. softplus ( input , beta = 1 , threshold = 20 ) [source] # tensorplay.softshrink URL: https://www.tensorplay.cn/docs/generated/tensorplay.softshrink.html tensorplay.softshrink tensorplay. softshrink ( input , lambd = 0.5 ) [source] # tensorplay.sparse_add URL: https://www.tensorplay.cn/docs/generated/tensorplay.sparse_add.html tensorplay.sparse_add tensorplay. sparse_add ( input , other ) [source] # tensorplay.sparse_mm URL: https://www.tensorplay.cn/docs/generated/tensorplay.sparse_mm.html tensorplay.sparse_mm tensorplay. sparse_mm ( input , dense ) [source] # tensorplay.sparse_mul URL: https://www.tensorplay.cn/docs/generated/tensorplay.sparse_mul.html tensorplay.sparse_mul tensorplay. sparse_mul ( input , other ) [source] # tensorplay.sparse_sum URL: https://www.tensorplay.cn/docs/generated/tensorplay.sparse_sum.html tensorplay.sparse_sum tensorplay. sparse_sum ( input , dim = None , dtype = None ) [source] # tensorplay.sparse.mm URL: https://www.tensorplay.cn/docs/generated/tensorplay.sparse.mm.html tensorplay.sparse.mm tensorplay.sparse. mm ( sparse , dense ) [source] Performs a matrix multiplication of a 2-D sparse COO/CSR tensor with a dense matrix. Equivalent to torch.sparse.mm . # tensorplay.sparse.spdiags URL: https://www.tensorplay.cn/docs/generated/tensorplay.sparse.spdiags.html tensorplay.sparse.spdiags tensorplay.sparse. spdiags ( diagonals , offsets , shape , layout = None ) [source] Constructs a sparse tensor from diagonals, mirroring torch.sparse.spdiags (and thereby scipy.sparse.spdiags ). Parameters : diagonals – matrix of shape (len(offsets), L) (or a single vector); row j holds the values of diagonal offsets[j] , read starting at column max(offset, 0) . offsets – int64 sequence of distinct diagonal offsets (0 = main, positive = above, negative = below). shape – 2-element (M, N) output size. layout – output layout tag — tensorplay.sparse_coo (default) or tensorplay.sparse_csr . # tensorplay.sparse.sum URL: https://www.tensorplay.cn/docs/generated/tensorplay.sparse.sum.html tensorplay.sparse.sum tensorplay.sparse. sum ( input , dim = None , dtype = None ) [source] Sum of input ’s values over dim . Mirrors torch.sparse.sum : with no dim (or an empty list) the result is a dense 0-dim tensor; reducing every sparse dim yields a dense tensor; a partial reduction returns a coalesced sparse COO tensor over the remaining dims with duplicate coordinates folded. dtype converts the input first, acting as the accumulation type. # tensorplay.spdiags URL: https://www.tensorplay.cn/docs/generated/tensorplay.spdiags.html tensorplay.spdiags tensorplay. spdiags ( diagonals , offsets , shape , layout = None ) [source] # tensorplay.special.entr URL: https://www.tensorplay.cn/docs/generated/tensorplay.special.entr.html tensorplay.special.entr tensorplay.special. entr ( input ) [source] Entropy of a probability element: -x*log(x) (0 at x=0, NaN below). # tensorplay.special.erfcx URL: https://www.tensorplay.cn/docs/generated/tensorplay.special.erfcx.html tensorplay.special.erfcx tensorplay.special. erfcx ( input ) [source] Scaled complementary error function: exp(x^2) * erfc(x) . # tensorplay.special.expit URL: https://www.tensorplay.cn/docs/generated/tensorplay.special.expit.html tensorplay.special.expit tensorplay.special. expit ( input ) [source] Numerically-stable sigmoid; identical to sigmoid() . # tensorplay.special.gammaln URL: https://www.tensorplay.cn/docs/generated/tensorplay.special.gammaln.html tensorplay.special.gammaln tensorplay.special. gammaln ( input ) [source] Natural log of the absolute value of the gamma function (native lgamma). # tensorplay.special.log_ndtr URL: https://www.tensorplay.cn/docs/generated/tensorplay.special.log_ndtr.html tensorplay.special.log_ndtr tensorplay.special. log_ndtr ( input ) [source] log of the standard normal CDF, stable in the left tail. # tensorplay.special.log_softmax URL: https://www.tensorplay.cn/docs/generated/tensorplay.special.log_softmax.html tensorplay.special.log_softmax tensorplay.special. log_softmax ( input , dim = None , * , dtype = None ) [source] # tensorplay.special.logsumexp URL: https://www.tensorplay.cn/docs/generated/tensorplay.special.logsumexp.html tensorplay.special.logsumexp tensorplay.special. logsumexp ( input , dim , keepdim = False , * , out = None ) [source] Log-sum-exp along dim , computed with the max-shift trick. # tensorplay.special.modified_bessel_i0 URL: https://www.tensorplay.cn/docs/generated/tensorplay.special.modified_bessel_i0.html tensorplay.special.modified_bessel_i0 tensorplay.special. modified_bessel_i0 ( input ) [source] Modified Bessel function of the first kind, order 0; alias of i0() . # tensorplay.special.multigammaln URL: https://www.tensorplay.cn/docs/generated/tensorplay.special.multigammaln.html tensorplay.special.multigammaln tensorplay.special. multigammaln ( input , p ) [source] Multivariate log-gamma with dimension p (p >= 2). # tensorplay.special.ndtr URL: https://www.tensorplay.cn/docs/generated/tensorplay.special.ndtr.html tensorplay.special.ndtr tensorplay.special. ndtr ( input ) [source] Standard normal CDF: Phi(x) = 0.5 * erfc(-x / sqrt(2)) . # tensorplay.special.ndtri URL: https://www.tensorplay.cn/docs/generated/tensorplay.special.ndtri.html tensorplay.special.ndtri tensorplay.special. ndtri ( input ) [source] Inverse standard normal CDF: sqrt(2) * erfinv(2x - 1) . # tensorplay.special.polygamma URL: https://www.tensorplay.cn/docs/generated/tensorplay.special.polygamma.html tensorplay.special.polygamma tensorplay.special. polygamma ( n , input ) [source] Polygamma of order :attr:n: \(\psi^{(n)}(x)\) (native kernel). # tensorplay.special.psi URL: https://www.tensorplay.cn/docs/generated/tensorplay.special.psi.html tensorplay.special.psi tensorplay.special. psi ( input ) [source] Digamma function; alias of digamma() . # tensorplay.special.round URL: https://www.tensorplay.cn/docs/generated/tensorplay.special.round.html tensorplay.special.round tensorplay.special. round ( input ) [source] Rounds to nearest even integer (native Tensor.round ). # tensorplay.special.softmax URL: https://www.tensorplay.cn/docs/generated/tensorplay.special.softmax.html tensorplay.special.softmax tensorplay.special. softmax ( input , dim = None , * , dtype = None ) [source] # tensorplay.special.xlog1py URL: https://www.tensorplay.cn/docs/generated/tensorplay.special.xlog1py.html tensorplay.special.xlog1py tensorplay.special. xlog1py ( input , other ) [source] x*log1p(y) with the convention x*log1p(-1) = 0 when x == 0. # tensorplay.special.xlogy URL: https://www.tensorplay.cn/docs/generated/tensorplay.special.xlogy.html tensorplay.special.xlogy tensorplay.special. xlogy ( input , other ) [source] x*log(y) with the convention 0*log(anything) = 0 . # tensorplay.spherical_bessel_j0 URL: https://www.tensorplay.cn/docs/generated/tensorplay.spherical_bessel_j0.html tensorplay.spherical_bessel_j0 tensorplay. spherical_bessel_j0 ( input ) [source] # tensorplay.split_with_sizes URL: https://www.tensorplay.cn/docs/generated/tensorplay.split_with_sizes.html tensorplay.split_with_sizes tensorplay. split_with_sizes ( input , split_sizes , dim = 0 ) [source] # tensorplay.stax.is_available URL: https://www.tensorplay.cn/docs/generated/tensorplay.stax.is_available.html tensorplay.stax.is_available tensorplay.stax. is_available ( ) → bool [source] Return whether the native Stax extension is loaded. # tensorplay.stax.stax URL: https://www.tensorplay.cn/docs/generated/tensorplay.stax.stax.html tensorplay.stax.stax tensorplay.stax. stax ( graph_module : GraphModule , example_inputs : list [ Any ] , * , mode : str | None = None , options : dict [ str , Any ] | None = None , name : str | None = None , dynamic : bool | None = None , strict_native : bool = False , ** kwargs : Any ) [source] Compile one canonical graph and return an executable callable. example_inputs and backend options are part of the same contract as TorchInductor’s compile_fx entry point. Stax currently specializes metadata in the frontend and uses the native graph when its lowering contract is satisfied. strict_native makes a failed lowering a hard compiler error, so a benchmark can never report the Python GraphModule executor as compiled performance. # tensorplay.sub_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.sub_.html tensorplay.sub_ tensorplay. sub_ ( input , other , alpha = 1 ) [source] # tensorplay.tensor URL: https://www.tensorplay.cn/docs/generated/tensorplay.tensor.html tensorplay.tensor tensorplay. tensor ( data : object , * , dtype : tensorplay._C.DType | None = None , device : tensorplay._C.Device | None = None , pin_memory : bool = False , requires_grad : bool = False ) → tensorplay._C.TensorBase # Tensor URL: https://www.tensorplay.cn/docs/generated/tensorplay.Tensor.html Tensor tensorplay. Tensor alias of TensorBase # tensorplay.to_dense URL: https://www.tensorplay.cn/docs/generated/tensorplay.to_dense.html tensorplay.to_dense tensorplay. to_dense ( input ) [source] # tensorplay.to_sparse_csr URL: https://www.tensorplay.cn/docs/generated/tensorplay.to_sparse_csr.html tensorplay.to_sparse_csr tensorplay. to_sparse_csr ( input ) [source] # tensorplay.to_sparse URL: https://www.tensorplay.cn/docs/generated/tensorplay.to_sparse.html tensorplay.to_sparse tensorplay. to_sparse ( input ) [source] # tensorplay.tp_binary_cross_entropy URL: https://www.tensorplay.cn/docs/generated/tensorplay.tp_binary_cross_entropy.html tensorplay.tp_binary_cross_entropy tensorplay. tp_binary_cross_entropy ( input , target , weight = None , reduction = 1 ) [source] # tensorplay.tp_cosine_embedding_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.tp_cosine_embedding_loss.html tensorplay.tp_cosine_embedding_loss tensorplay. tp_cosine_embedding_loss ( input1 , input2 , target , margin = 0.0 , reduction = 1 ) [source] # tensorplay.tp_hinge_embedding_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.tp_hinge_embedding_loss.html tensorplay.tp_hinge_embedding_loss tensorplay. tp_hinge_embedding_loss ( input , target , margin = 1.0 , reduction = 1 ) [source] # tensorplay.tp_huber_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.tp_huber_loss.html tensorplay.tp_huber_loss tensorplay. tp_huber_loss ( input , target , reduction = 1 , delta = 1.0 ) [source] # tensorplay.tp_kl_div URL: https://www.tensorplay.cn/docs/generated/tensorplay.tp_kl_div.html tensorplay.tp_kl_div tensorplay. tp_kl_div ( input , target , reduction = 1 , log_target = False ) [source] # tensorplay.tp_l1_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.tp_l1_loss.html tensorplay.tp_l1_loss tensorplay. tp_l1_loss ( input , target , reduction = 1 ) [source] # tensorplay.tp_margin_ranking_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.tp_margin_ranking_loss.html tensorplay.tp_margin_ranking_loss tensorplay. tp_margin_ranking_loss ( input1 , input2 , target , margin = 0.0 , reduction = 1 ) [source] # tensorplay.tp_poisson_nll_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.tp_poisson_nll_loss.html tensorplay.tp_poisson_nll_loss tensorplay. tp_poisson_nll_loss ( input , target , log_input = True , full = False , eps = 1e-08 , reduction = 1 ) [source] # tensorplay.tp_smooth_l1_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.tp_smooth_l1_loss.html tensorplay.tp_smooth_l1_loss tensorplay. tp_smooth_l1_loss ( input , target , reduction = 1 , beta = 1.0 ) [source] # tensorplay.tp_soft_margin_loss URL: https://www.tensorplay.cn/docs/generated/tensorplay.tp_soft_margin_loss.html tensorplay.tp_soft_margin_loss tensorplay. tp_soft_margin_loss ( input , target , reduction = 1 ) [source] # tensorplay.typename URL: https://www.tensorplay.cn/docs/generated/tensorplay.typename.html tensorplay.typename tensorplay. typename ( obj : Any , / ) → str [source] String representation of the type of an object. This function returns a fully qualified string representation of an object’s type. :param obj: The object whose type to represent :type obj: object Returns : the type of the object o Return type : str Example >>> x = tensorplay . tensor ([ 1 , 2 , 3 ]) >>> tensorplay . typename ( x ) 'tensorplay.LongTensor' >>> tensorplay . typename ( tensorplay . nn . Parameter ) 'tensorplay.nn.parameter.Parameter' # tensorplay.unfold URL: https://www.tensorplay.cn/docs/generated/tensorplay.unfold.html tensorplay.unfold tensorplay. unfold ( input , dimension , size , step ) [source] # tensorplay.uniform_ URL: https://www.tensorplay.cn/docs/generated/tensorplay.uniform_.html tensorplay.uniform_ tensorplay. uniform_ ( input , from_ = 0.0 , to = 1.0 ) [source] # tensorplay.unsafe_chunk URL: https://www.tensorplay.cn/docs/generated/tensorplay.unsafe_chunk.html tensorplay.unsafe_chunk tensorplay. unsafe_chunk ( input , chunks , dim = 0 ) [source] # tensorplay.unsafe_split URL: https://www.tensorplay.cn/docs/generated/tensorplay.unsafe_split.html tensorplay.unsafe_split tensorplay. unsafe_split ( input , split_size , dim = 0 ) [source] # tensorplay.upsample_bicubic2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.upsample_bicubic2d.html tensorplay.upsample_bicubic2d tensorplay. upsample_bicubic2d ( input , output_size , align_corners , scales_h = None , scales_w = None ) [source] # tensorplay.upsample_bilinear2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.upsample_bilinear2d.html tensorplay.upsample_bilinear2d tensorplay. upsample_bilinear2d ( input , output_size , align_corners , scales_h = None , scales_w = None ) [source] # tensorplay.upsample_linear1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.upsample_linear1d.html tensorplay.upsample_linear1d tensorplay. upsample_linear1d ( input , output_size , align_corners , scales = None ) [source] # tensorplay.upsample_nearest1d URL: https://www.tensorplay.cn/docs/generated/tensorplay.upsample_nearest1d.html tensorplay.upsample_nearest1d tensorplay. upsample_nearest1d ( input , output_size , scales = None ) [source] # tensorplay.upsample_nearest2d URL: https://www.tensorplay.cn/docs/generated/tensorplay.upsample_nearest2d.html tensorplay.upsample_nearest2d tensorplay. upsample_nearest2d ( input , output_size , scales_h = None , scales_w = None ) [source] # tensorplay.upsample_nearest3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.upsample_nearest3d.html tensorplay.upsample_nearest3d tensorplay. upsample_nearest3d ( input , output_size , scales_d = None , scales_h = None , scales_w = None ) [source] # tensorplay.upsample_trilinear3d URL: https://www.tensorplay.cn/docs/generated/tensorplay.upsample_trilinear3d.html tensorplay.upsample_trilinear3d tensorplay. upsample_trilinear3d ( input , output_size , align_corners , scales_d = None , scales_h = None , scales_w = None ) [source] # tensorplay.use_deterministic_algorithms URL: https://www.tensorplay.cn/docs/generated/tensorplay.use_deterministic_algorithms.html tensorplay.use_deterministic_algorithms tensorplay. use_deterministic_algorithms ( mode : bool , * , warn_only : bool = False ) → None [source] Sets whether TensorPlay operations must use “deterministic” algorithms. That is, algorithms which, given the same input, and when run on the same software and hardware, always produce the same output. When enabled, operations will use deterministic algorithms when available, and if only nondeterministic algorithms are available they will throw a RuntimeError when called. Note This setting alone is not always enough to make an application reproducible. Refer to Reproducibility for more information. Note tensorplay.set_deterministic_debug_mode() offers an alternative interface for this feature. Note that deterministic operations tend to have worse performance than nondeterministic operations. Note This flag does not detect or prevent nondeterministic behavior caused by calling an inplace operation on a tensor with an internal memory overlap or by giving such a tensor as the out argument for an operation. In these cases, multiple writes of different data may target a single memory location, and the order of writes is not guaranteed. Parameters : mode ( bool ) – If True, makes potentially nondeterministic operations switch to a deterministic algorithm or throw a runtime error. If False, allows nondeterministic operations. Keyword Arguments : warn_only ( bool , optional) – If True, operations that do not have a deterministic implementation will throw a warning instead of an error. Default: False Example: >>> # xdoctest: +SKIP >>> tensorplay . use_deterministic_algorithms ( True ) # tensorplay.utils.checkpoint.checkpoint_sequential URL: https://www.tensorplay.cn/docs/generated/tensorplay.utils.checkpoint.checkpoint_sequential.html tensorplay.utils.checkpoint.checkpoint_sequential tensorplay.utils.checkpoint. checkpoint_sequential ( function : Callable [ [ ... ] , Any ] , chunks : int , * args : Any , ** kwargs : Any ) → Any [source] torch.utils.checkpoint.checkpoint_sequential compatibility. With chunks segments the sequential would be evaluated in chunks with recomputation; TensorPlay runs it as one eager segment. # tensorplay.utils.checkpoint.checkpoint URL: https://www.tensorplay.cn/docs/generated/tensorplay.utils.checkpoint.checkpoint.html tensorplay.utils.checkpoint.checkpoint tensorplay.utils.checkpoint. checkpoint ( function : Callable [ [ ... ] , Any ] , * args : Any , use_reentrant : bool = True , context_fn : Callable [ [ ] , Any ] | None = None , determinism_check : str = 'default' , debug : bool = False , ** kwargs : Any ) → Any [source] Runs function(*args, **kwargs) eagerly (no recomputation). Mirrors torch.utils.checkpoint.checkpoint’s signature; emits a warning once when grad is enabled because backward will re-run the graph normally. # CheckpointPolicy URL: https://www.tensorplay.cn/docs/generated/tensorplay.utils.checkpoint.CheckpointPolicy.html CheckpointPolicy class tensorplay.utils.checkpoint. CheckpointPolicy [source] # tensorplay.utils.checkpoint.set_checkpoint_early_stop URL: https://www.tensorplay.cn/docs/generated/tensorplay.utils.checkpoint.set_checkpoint_early_stop.html tensorplay.utils.checkpoint.set_checkpoint_early_stop tensorplay.utils.checkpoint. set_checkpoint_early_stop ( enabled : bool ) → None [source] No-op for API parity. # BatchSampler URL: https://www.tensorplay.cn/docs/generated/tensorplay.utils.data.BatchSampler.html BatchSampler class tensorplay.utils.data. BatchSampler ( sampler : Sampler [ int ] | Iterable [ int ] , batch_size : int , drop_last : bool ) [source] Wraps another sampler to yield a mini-batch of indices. Parameters : sampler ( Sampler or Iterable ) – Base sampler. Can be any iterable object batch_size ( int ) – Size of mini-batch. drop_last ( bool ) – If True , the sampler will drop the last batch if its size would be less than batch_size # ChainDataset URL: https://www.tensorplay.cn/docs/generated/tensorplay.utils.data.ChainDataset.html ChainDataset class tensorplay.utils.data. ChainDataset ( datasets : Iterable [ Dataset ] ) [source] Dataset for chaining multiple IterableDataset s. This class is useful to assemble different existing dataset streams. The chaining operation is done on-the-fly, so concatenating large-scale datasets with this class will be efficient. Parameters : datasets ( iterable of IterableDataset ) – datasets to be chained together # ConcatDataset URL: https://www.tensorplay.cn/docs/generated/tensorplay.utils.data.ConcatDataset.html ConcatDataset class tensorplay.utils.data. ConcatDataset ( datasets : Iterable [ Dataset ] ) [source] Dataset as a concatenation of multiple datasets. This class is useful to assemble different existing datasets. Parameters : datasets ( sequence ) – List of datasets to be concatenated # DataLoader URL: https://www.tensorplay.cn/docs/generated/tensorplay.utils.data.DataLoader.html DataLoader class tensorplay.utils.data. DataLoader ( dataset : Dataset [ _T_co ] , batch_size : int | None = 1 , shuffle : bool | None = None , sampler : Sampler | Iterable | None = None , batch_sampler : Sampler | Iterable | None = None , num_workers : int = 0 , collate_fn : Callable [ [ List [ Any ] ] , Any ] | None = None , pin_memory : bool = False , drop_last : bool = False , timeout : float = 0 , worker_init_fn : Callable [ [ int ] , None ] | None = None , multiprocessing_context = None , generator : Generator | None = None , * , prefetch_factor : int | None = None , persistent_workers : bool = False , in_order : bool = True , device : str | None = None ) [source] Data loader combines a dataset and a sampler, and provides an iterable over the given dataset. The DataLoader supports both map-style and iterable-style datasets with single- or multi-process loading, customizing loading order and optional automatic batching (collation). Parameters : dataset ( Dataset ) – dataset from which to load the data. batch_size ( int , optional ) – how many samples per batch to load (default: 1 ). shuffle ( bool , optional ) – set to True to have the data reshuffled at every epoch (default: False ). sampler ( Sampler or Iterable , optional ) – defines the strategy to draw samples from the dataset. Can be any Iterable with __len__ implemented. If specified, shuffle must not be specified. batch_sampler ( Sampler or Iterable , optional ) – like sampler , but returns a batch of indices at a time. Mutually exclusive with batch_size , shuffle , sampler , and drop_last . num_workers ( int , optional ) – how many subprocesses to use for data loading. 0 means that the data will be loaded in the main process. (default: 0 ) collate_fn ( Callable , optional ) – merges a list of samples to form a mini-batch of Tensor(s). Used when using batched loading from a map-style dataset. pin_memory ( bool , optional ) – If True , the data loader will copy Tensors into CUDA page-locked host memory before returning them. drop_last ( bool , optional ) – set to True to drop the last incomplete batch, if the dataset size is not divisible by the batch size. (default: False ) timeout ( numeric , optional ) – if positive, the timeout value for collecting a batch from workers. Should always be non-negative. (default: 0 ) worker_init_fn ( Callable , optional ) – If not None , this will be called on each worker subprocess with the worker id (an int in [0, num_workers - 1] ) as input, before data loading. (default: None ) multiprocessing_context ( str or context , optional ) – start method or multiprocessing context used to spawn the workers, e.g., "fork" or "spawn" . If None , the default context of the platform is used. (default: None ) generator ( Generator , optional ) – If not None , this RNG will be used by RandomSampler to generate random indexes. (default: None ) prefetch_factor ( int , optional ) – Number of batches loaded in advance by each worker. 2 means there will be a total of 2 * num_workers batches prefetched across all workers. (default: 2 when num_workers > 0 ; otherwise must be None ) persistent_workers ( bool , optional ) – If True , the data loader will not shut down the worker processes after a dataset has been consumed once. This allows to maintain the workers Dataset instances alive. (default: False ) in_order ( bool , optional ) – If False , the data loader will not enforce that batches returned from multiprocessing workers are provided in the order the sampler produced them. This enables faster delivery of batches that complete early, at the cost of batch order no longer being deterministic. (default: True ) device ( str , optional ) – device to move batches to after collation. # Dataset URL: https://www.tensorplay.cn/docs/generated/tensorplay.utils.data.Dataset.html Dataset class tensorplay.utils.data. Dataset [source] An abstract class representing a Dataset . All datasets that represent a map from keys to data samples should subclass it. All subclasses should overwrite __getitem__() , supporting fetching a data sample for a given key. Subclasses could also optionally overwrite __len__() , which is expected to return the size of the dataset by many Sampler implementations and the default options of DataLoader . Note DataLoader by default constructs an index sampler that yields integral indices. To make it work with a map-style dataset with non-integral indices/keys, a custom sampler must be provided. # tensorplay.utils.data.default_collate URL: https://www.tensorplay.cn/docs/generated/tensorplay.utils.data.default_collate.html tensorplay.utils.data.default_collate tensorplay.utils.data. default_collate ( batch ) [source] Take in a batch of data and put the elements within the batch into a tensor with an additional outer dimension - batch size. The exact output type can be a tensorplay.Tensor , a Sequence of tensorplay.Tensor , a Collection of tensorplay.Tensor , or left unchanged, depending on the input type. This is used as the default function for collation when batch_size or batch_sampler is defined in DataLoader . Parameters : batch – a single batch to be collated # tensorplay.utils.data.default_convert URL: https://www.tensorplay.cn/docs/generated/tensorplay.utils.data.default_convert.html tensorplay.utils.data.default_convert tensorplay.utils.data. default_convert ( data ) [source] Convert each NumPy array element into a tensorplay.Tensor . If the input is a Sequence , Collection , or Mapping , it tries to convert each element inside to a tensorplay.Tensor . If the input is not a NumPy array, it is left unchanged. This is used as the default function for collation when both batch_sampler and batch_size are NOT defined in DataLoader . Parameters : data – a single data point to be converted # tensorplay.utils.data.get_worker_info URL: https://www.tensorplay.cn/docs/generated/tensorplay.utils.data.get_worker_info.html tensorplay.utils.data.get_worker_info tensorplay.utils.data. get_worker_info ( ) → WorkerInfo | None [source] Returns the information about the current DataLoader iterator worker process. When called in a worker process, returns a WorkerInfo object with information about that worker process; otherwise returns None . # IterableDataset URL: https://www.tensorplay.cn/docs/generated/tensorplay.utils.data.IterableDataset.html IterableDataset class tensorplay.utils.data. IterableDataset [source] An iterable Dataset. All datasets that represent an iterable of data samples should subclass it. Such form of datasets is particularly useful when data come from a stream. All subclasses should overwrite __iter__() , which would return an iterator of samples in this dataset. When a subclass is used with DataLoader , each item in the dataset will be yielded from the DataLoader iterator. # RandomSampler URL: https://www.tensorplay.cn/docs/generated/tensorplay.utils.data.RandomSampler.html RandomSampler class tensorplay.utils.data. RandomSampler ( data_source : Sized , replacement : bool = False , num_samples : int | None = None , generator : Generator | None = None ) [source] Samples elements randomly. If without replacement, then sample from a shuffled dataset. If with replacement, then user can specify num_samples to draw. Parameters : data_source ( Sized ) – data source to sample from. Must implement __len__. replacement ( bool ) – samples are drawn on-demand with replacement if True , default=``False``. num_samples ( int ) – number of samples to draw, default=`len(dataset)`. generator ( Generator ) – Generator used in sampling. # Sampler URL: https://www.tensorplay.cn/docs/generated/tensorplay.utils.data.Sampler.html Sampler class tensorplay.utils.data. Sampler [source] Base class for all Samplers. Every Sampler subclass has to provide an __iter__() method, providing a way to iterate over indices or lists of indices (batches) of dataset elements, and may provide a __len__() method that returns the length of the returned iterators. Note The __len__() method isn’t strictly required by DataLoader , but is expected in any calculation involving the length of a DataLoader. # SequentialSampler URL: https://www.tensorplay.cn/docs/generated/tensorplay.utils.data.SequentialSampler.html SequentialSampler class tensorplay.utils.data. SequentialSampler ( data_source : Sized ) [source] Samples elements sequentially, always in the same order. Parameters : data_source ( Sized ) – data source to sample from. Must implement __len__. # StackDataset URL: https://www.tensorplay.cn/docs/generated/tensorplay.utils.data.StackDataset.html StackDataset class tensorplay.utils.data. StackDataset ( * args : Dataset [ _T_co ] , ** kwargs : Dataset [ _T_co ] ) [source] Dataset as a stacking of multiple datasets. This class is useful to assemble different parts of complex input data, given as datasets. Example >>> images = ImageDataset () >>> texts = TextDataset () >>> tuple_stack = StackDataset ( images , texts ) >>> tuple_stack [ 0 ] == ( images [ 0 ], texts [ 0 ]) >>> dict_stack = StackDataset ( image = images , text = texts ) >>> dict_stack [ 0 ] == { "image" : images [ 0 ], "text" : texts [ 0 ]} Parameters : *args ( Dataset ) – Datasets for stacking returned as tuple. **kwargs ( Dataset ) – Datasets for stacking returned as dict. # Subset URL: https://www.tensorplay.cn/docs/generated/tensorplay.utils.data.Subset.html Subset class tensorplay.utils.data. Subset ( dataset : Dataset [ _T_co ] , indices : Sequence [ int ] ) [source] Subset of a dataset at specified indices. Note When subclassing Subset and overriding __getitem__ , you must also override __getitems__ to ensure DataLoader works correctly with your custom logic. If you override only __getitem__ , a NotImplementedError will be raised when using DataLoader . Parameters : dataset ( Dataset ) – The whole Dataset indices ( sequence ) – Indices in the whole set selected for subset # SubsetRandomSampler URL: https://www.tensorplay.cn/docs/generated/tensorplay.utils.data.SubsetRandomSampler.html SubsetRandomSampler class tensorplay.utils.data. SubsetRandomSampler ( indices : Sequence [ int ] , generator : Generator | None = None ) [source] Samples elements randomly from a given list of indices, without replacement. Parameters : indices ( sequence ) – a sequence of indices generator ( Generator ) – Generator used in sampling. # TensorDataset URL: https://www.tensorplay.cn/docs/generated/tensorplay.utils.data.TensorDataset.html TensorDataset class tensorplay.utils.data. TensorDataset ( * tensors : TensorBase ) [source] Dataset wrapping tensors. Each sample will be retrieved by indexing tensors along the first dimension. Parameters : *tensors ( Tensor ) – tensors that have the same size of the first dimension. # WeightedRandomSampler URL: https://www.tensorplay.cn/docs/generated/tensorplay.utils.data.WeightedRandomSampler.html WeightedRandomSampler class tensorplay.utils.data. WeightedRandomSampler ( weights : Sequence [ float ] , num_samples : int , replacement : bool = True , generator : Generator | None = None ) [source] Samples elements from [0,..,len(weights)-1] with given probabilities (weights). Parameters : weights ( sequence ) – a sequence of weights, not necessary summing up to one num_samples ( int ) – number of samples to draw replacement ( bool ) – if True , samples are drawn with replacement. If not, they are drawn without replacement, which means that when a sample index is drawn for a row, it cannot be drawn again for that row. generator ( Generator ) – Generator used in sampling. # tensorplay.view URL: https://www.tensorplay.cn/docs/generated/tensorplay.view.html tensorplay.view tensorplay. view ( input , shape ) [source] # tensorplay.zeta URL: https://www.tensorplay.cn/docs/generated/tensorplay.zeta.html tensorplay.zeta tensorplay. zeta ( input , other ) [source] # tensorplay.hub URL: https://www.tensorplay.cn/docs/hub.html tensorplay.hub Loading models from Hub tensorplay.hub.load Unified model loader. tensorplay.hub.download_url_to_file Download a URL to a local file(Safe download: temp file + hash check + progress feedback). tensorplay.hub.load_state_dict_from_url Downloads a raw checkpoint URL and loads it (torch semantics). Where are my downloaded models saved? tensorplay.hub.get_dir Get the TensorPlay Hub cache directory used for storing downloaded models & weights. tensorplay.hub.set_dir Optionally set the TensorPlay Hub directory used to save downloaded models & weights. # tensorplay.library URL: https://www.tensorplay.cn/docs/library.html tensorplay.library Choosing the kind of custom op tensorplay.library.custom_op Define a user operator; use as a decorator (torch parity). tensorplay.library.triton_op Define a Triton-backed operator ( torch.library.triton_op ). tensorplay.library.wrap_triton Mark a Triton kernel as launchable from within a triton_op . Extending custom ops (created from Python or C++) tensorplay.library.register_kernel Top-level kernel registration ( torch.library.register_kernel ). tensorplay.library.register_autograd Top-level autograd registration ( torch.library.register_autograd ). tensorplay.library.register_fake Top-level fake-kernel registration ( torch.library.register_fake ). Low-level APIs tensorplay.library.Library Scoped registration bundle ( torch.library.Library subset). # tensorplay.linalg URL: https://www.tensorplay.cn/docs/linalg.html tensorplay.linalg Matrix Properties tensorplay.linalg.norm tensorplay.linalg.vector_norm tensorplay.linalg.matrix_norm tensorplay.diagonal tensorplay.linalg.det tensorplay.linalg.slogdet tensorplay.linalg.cond tensorplay.linalg.matrix_rank Decompositions tensorplay.linalg.cholesky tensorplay.linalg.qr tensorplay.linalg.polar tensorplay.linalg.lu tensorplay.linalg.lu_factor tensorplay.linalg.eig tensorplay.linalg.eigvals tensorplay.linalg.eigh tensorplay.linalg.eigvalsh tensorplay.linalg.svd tensorplay.linalg.svdvals Solvers tensorplay.linalg.solve tensorplay.linalg.solve_triangular tensorplay.linalg.lu_solve tensorplay.linalg.lstsq Inverses tensorplay.linalg.inv tensorplay.linalg.pinv Matrix Functions tensorplay.linalg.matrix_exp Square matrix exponential via Pade approximation with scaling and squaring (the algorithm of torch's linalg_matrix_exp, Higham 2005). tensorplay.linalg.matrix_sqrth Matrix square root via the Denman-Beavers fixed-point iteration (converges for matrices with no eigenvalues on the closed negative real axis). tensorplay.linalg.matrix_power Matrix Products tensorplay.linalg.cross tensorplay.linalg.matmul tensorplay.linalg.vecdot tensorplay.linalg.multi_dot tensorplay.linalg.householder_product Tensor Operations tensorplay.linalg.tensorinv tensorplay.linalg.tensorsolve Solves A X = B where A and B are (tuples of) matrices interpreted as a single square system over the trailing dimensions. Misc tensorplay.linalg.vander Experimental Functions tensorplay.linalg.cholesky_ex tensorplay.linalg.inv_ex tensorplay.linalg.solve_ex tensorplay.linalg.lu_factor_ex tensorplay.linalg.ldl_factor tensorplay.linalg.ldl_factor_ex tensorplay.linalg.ldl_solve # Multiprocessing package - tensorplay.multiprocessing URL: https://www.tensorplay.cn/docs/multiprocessing.html Multiprocessing package - tensorplay.multiprocessing API Reference tensorplay.multiprocessing.reduce_tensor tensorplay.multiprocessing.allow_connection_pickling Install support for sending connections and sockets between processes tensorplay.multiprocessing.set_start_method tensorplay.multiprocessing.get_start_method tensorplay.multiprocessing.get_all_start_methods # tensorplay.nn.functional URL: https://www.tensorplay.cn/docs/nn.functional.html tensorplay.nn.functional Convolution functions tensorplay.nn.functional.conv1d Applies a 1D convolution over an input signal composed of several input planes. tensorplay.nn.functional.conv2d Applies a 2D convolution over an input image composed of several input planes. tensorplay.nn.functional.conv3d Applies a 3D convolution over an input image composed of several input planes. tensorplay.nn.functional.conv_transpose1d tensorplay.nn.functional.conv_transpose2d tensorplay.nn.functional.conv_transpose3d tensorplay.nn.functional.unfold Extract sliding local blocks from a batched input tensor (torch F.unfold, i.e. aten::im2col). tensorplay.nn.functional.fold Combine an array of sliding local blocks into a tensor containing them all (torch F.fold, i.e. aten::col2im). Pooling functions tensorplay.nn.functional.avg_pool1d avg_pool1d(input, kernel_size, stride=None, padding=0, ceil_mode=False, count_include_pad=True, divisor_override=None) -> Tensor tensorplay.nn.functional.avg_pool2d tensorplay.nn.functional.avg_pool3d avg_pool3d(input, kernel_size, stride=None, padding=0, ceil_mode=False, count_include_pad=True, divisor_override=None) -> Tensor tensorplay.nn.functional.max_pool1d max_pool1d(input, kernel_size, stride=None, padding=0, dilation=1, ceil_mode=False, return_indices=False) -> Tensor tensorplay.nn.functional.max_pool2d tensorplay.nn.functional.max_pool3d max_pool3d(input, kernel_size, stride=None, padding=0, dilation=1, ceil_mode=False, return_indices=False) -> Tensor tensorplay.nn.functional.max_unpool1d Compute a partial inverse of MaxPool1d . tensorplay.nn.functional.max_unpool2d Compute a partial inverse of MaxPool2d . tensorplay.nn.functional.max_unpool3d Compute a partial inverse of MaxPool3d . tensorplay.nn.functional.lp_pool1d Apply a 1D power-average pooling over an input signal. tensorplay.nn.functional.lp_pool2d Apply a 2D power-average pooling over an input signal. tensorplay.nn.functional.lp_pool3d Apply a 3D power-average pooling over an input signal. tensorplay.nn.functional.adaptive_max_pool1d tensorplay.nn.functional.adaptive_max_pool2d tensorplay.nn.functional.adaptive_avg_pool1d tensorplay.nn.functional.adaptive_avg_pool2d tensorplay.nn.functional.adaptive_avg_pool3d Apply a 3D adaptive average pooling over an input signal. tensorplay.nn.functional.fractional_max_pool2d Applies 2D fractional max pooling over an input signal. tensorplay.nn.functional.fractional_max_pool3d Applies 3D fractional max pooling over an input signal. Attention Mechanisms tensorplay.nn.functional.scaled_dot_product_attention scaled_dot_product_attention(query, key, value, attn_mask=None, dropout_p=0.0, is_causal=False, scale=None, backend=None) -> Tensor Non-linear activation functions tensorplay.nn.functional.threshold Apply a threshold to each element of the input Tensor. tensorplay.nn.functional.relu tensorplay.nn.functional.hardtanh tensorplay.nn.functional.hardswish tensorplay.nn.functional.relu6 ReLU6: \(\min(\max(0, x), 6)\) — torch.nn.functional.relu6. tensorplay.nn.functional.elu tensorplay.nn.functional.selu tensorplay.nn.functional.celu tensorplay.nn.functional.leaky_relu tensorplay.nn.functional.prelu tensorplay.nn.functional.rrelu Randomized leaky ReLU. tensorplay.nn.functional.glu Gated Linear Unit: \(a * \sigma(b)\) where the input is split in half along dim . tensorplay.nn.functional.gelu When approximate is 'none', applies \(\text{GELU}(x) = x * \Phi(x)\) ; 'tanh' uses the tanh estimation. tensorplay.nn.functional.logsigmoid Applies element-wise \(\text{LogSigmoid}(x_i) = \log \left(\frac{1}{1 + \exp(-x_i)}\right)\) tensorplay.nn.functional.hardshrink Applies the hard shrinkage function element-wise. tensorplay.nn.functional.tanhshrink Applies element-wise, \(\text{Tanhshrink}(x) = x - \text{Tanh}(x)\) tensorplay.nn.functional.softsign Applies element-wise, the function \(\text{SoftSign}(x) = \frac{x}{1 + |x|}\) tensorplay.nn.functional.softplus tensorplay.nn.functional.softmin Apply a softmin function. tensorplay.nn.functional.softmax tensorplay.nn.functional.softshrink Applies the soft shrinkage function element-wise. tensorplay.nn.functional.gumbel_softmax Sample from the Gumbel-Softmax distribution and optionally discretize. tensorplay.nn.functional.log_softmax tensorplay.nn.functional.tanh Applies element-wise \(\text{Tanh}(x) = \frac{\exp(x) - \exp(-x)}{\exp(x) + \exp(-x)}\) tensorplay.nn.functional.sigmoid Applies the element-wise function \(\text{Sigmoid}(x) = \frac{1}{1 + \exp(-x)}\) tensorplay.nn.functional.hardsigmoid tensorplay.nn.functional.silu Apply the Sigmoid Linear Unit (SiLU) function, element-wise. tensorplay.nn.functional.mish tensorplay.nn.functional.batch_norm tensorplay.nn.functional.group_norm tensorplay.nn.functional.instance_norm tensorplay.nn.functional.layer_norm tensorplay.nn.functional.local_response_norm Apply local response normalization over an input signal. tensorplay.nn.functional.rms_norm Apply Root Mean Square Layer Normalization — composed per the ATen rms_norm composite (fp32 compute for reduced dtypes). tensorplay.nn.functional.normalize Performs \(L_p\) normalization over the specified dimension — torch.nn.functional.normalize divides by clamp_min(norm, eps). Linear functions tensorplay.nn.functional.linear Applies a linear transformation to the incoming data: \(y = xA^T + b\) . tensorplay.nn.functional.bilinear Dropout functions tensorplay.nn.functional.dropout tensorplay.nn.functional.alpha_dropout tensorplay.nn.functional.feature_alpha_dropout Randomly masks out entire channels, setting activations to the negative saturation value of the SELU activation function. tensorplay.nn.functional.dropout1d Randomly zero out entire channels (a channel is a 1D feature map). tensorplay.nn.functional.dropout2d tensorplay.nn.functional.dropout3d Sparse functions tensorplay.nn.functional.embedding tensorplay.nn.functional.embedding_bag Compute sums, means or maxes of bags of embeddings. tensorplay.nn.functional.one_hot Returns long tensor shaped tensor.shape + (num_classes,) with a 1 at each label position — port of ATen one_hot. Distance functions tensorplay.nn.functional.pairwise_distance Computes the pairwise distance between input vectors. tensorplay.nn.functional.cosine_similarity Returns cosine similarity between x1 and x2, computed along dim. tensorplay.nn.functional.pdist Computes the pairwise distance between rows of input . Loss functions tensorplay.nn.functional.binary_cross_entropy Compute Binary Cross Entropy between the target and input probabilities. tensorplay.nn.functional.binary_cross_entropy_with_logits Compute Binary Cross Entropy between target and input logits. tensorplay.nn.functional.poisson_nll_loss Compute the Poisson negative log likelihood loss. tensorplay.nn.functional.cosine_embedding_loss Compute the cosine embedding loss. tensorplay.nn.functional.cross_entropy Compute the cross entropy loss between input logits and target. tensorplay.nn.functional.ctc_loss Compute the Connectionist Temporal Classification loss. tensorplay.nn.functional.gaussian_nll_loss Compute the Gaussian negative log likelihood loss. tensorplay.nn.functional.hinge_embedding_loss Compute the hinge embedding loss. tensorplay.nn.functional.kl_div Compute the KL Divergence loss. tensorplay.nn.functional.l1_loss Compute the L1 loss, with optional weighting. tensorplay.nn.functional.linear_cross_entropy Compute cross entropy between input , transformed linearly, and target. tensorplay.nn.functional.mse_loss tensorplay.nn.functional.margin_ranking_loss Compute the margin ranking loss. tensorplay.nn.functional.multilabel_margin_loss Compute the multilabel margin loss. tensorplay.nn.functional.multilabel_soft_margin_loss Compute the multilabel soft margin loss. tensorplay.nn.functional.multi_margin_loss Compute the multi margin loss, with optional weighting. tensorplay.nn.functional.nll_loss The negative log likelihood loss. tensorplay.nn.functional.huber_loss Compute the Huber loss, with optional weighting. tensorplay.nn.functional.smooth_l1_loss Compute the Smooth L1 loss. tensorplay.nn.functional.soft_margin_loss Compute the soft margin loss. tensorplay.nn.functional.triplet_margin_loss Compute the triplet loss between given input tensors and a margin greater than 0. tensorplay.nn.functional.triplet_margin_with_distance_loss Compute the triplet margin loss using a custom distance function. Vision functions tensorplay.nn.functional.pixel_shuffle Rearranges elements in a tensor of shape (*, C x r^2, H, W) to a tensor of shape (*, C, H x r, W x r) . tensorplay.nn.functional.pixel_unshuffle Reverses the pixel_shuffle() transformation: (*, C, H x r, W x r) -> (*, C x r^2, H, W) . tensorplay.nn.functional.pad Pads tensor. tensorplay.nn.functional.interpolate interpolate(input, size=None, scale_factor=None, mode='nearest', align_corners=None) -> Tensor tensorplay.nn.functional.upsample Upsamples the input to the given size or scale_factor . tensorplay.nn.functional.upsample_nearest Upsamples the input using nearest neighbours. tensorplay.nn.functional.upsample_bilinear Upsamples the input using bilinear upsampling. tensorplay.nn.functional.grid_sample Compute grid sample. tensorplay.nn.functional.affine_grid Generate 2D or 3D flow field (sampling grid), given a batch of affine matrices theta . data_parallel Low-Precision functions tensorplay.nn.functional.grouped_mm tensorplay.nn.functional.scaled_mm tensorplay.nn.functional.scaled_grouped_mm TensorPlay-specific additions DType Members: Tensor alias of TensorBase adaptive_max_pool1d_with_indices Applies a 1D adaptive max pooling over an input signal, returning (output, indices) . adaptive_max_pool2d_with_indices Applies a 2D adaptive max pooling over an input signal composed of several input planes, returning (output, indices) . adaptive_max_pool3d_with_indices Applies a 3D adaptive max pooling over an input signal, returning (output, indices) . channel_shuffle Divide the channels in a tensor into g groups and rearrange them as in ShuffleNet: (*, C, H, W) -> (*, C, H, W) with channels interleaved across groups. conv_tbc Applies a 1D convolution over an input of shape (T, B, C) along the time dimension (torch.conv_tbc). flatten fractional_max_pool2d_with_indices Applies 2D fractional max pooling over an input signal composed of several input planes, returning (output, indices) . fractional_max_pool3d_with_indices Applies 3D fractional max pooling over an input signal composed of several input planes, returning (output, indices) . max_pool1d_with_indices Applies a 1D max pooling over an input signal, returning (output, indices) . max_pool2d_with_indices Applies a 2D max pooling over an input composed of several input planes, returning (output, indices) . max_pool3d_with_indices Applies a 3D max pooling over an input signal, returning (output, indices) . multi_head_attention_forward torch-compatible multi_head_attention_forward. native_channel_shuffle Native channel shuffle primitive (torch.native_channel_shuffle). # tensorplay.nn URL: https://www.tensorplay.cn/docs/nn.html tensorplay.nn Containers tensorplay.nn.modules.module.Module Base class for all neural network modules. tensorplay.nn.modules.container.Sequential A sequential container. tensorplay.nn.modules.container.ModuleList Holds submodules in a list. tensorplay.nn.modules.container.ModuleDict Holds submodules in a dictionary. tensorplay.nn.modules.container.ParameterList Holds parameters in a list. tensorplay.nn.modules.container.ParameterDict Holds parameters in a dictionary. tensorplay.nn.modules.module.register_module_forward_pre_hook Register a forward pre-hook common to all modules. tensorplay.nn.modules.module.register_module_forward_hook Register a global forward hook for all the modules. tensorplay.nn.modules.module.register_module_backward_hook Register a backward hook common to all the modules. tensorplay.nn.modules.module.register_module_full_backward_pre_hook Register a backward pre-hook common to all the modules. tensorplay.nn.modules.module.register_module_full_backward_hook Register a backward hook common to all the modules. tensorplay.nn.modules.module.register_module_buffer_registration_hook Register a buffer registration hook common to all modules. tensorplay.nn.modules.module.register_module_module_registration_hook Register a module registration hook common to all modules. tensorplay.nn.modules.module.register_module_parameter_registration_hook Register a parameter registration hook common to all modules. Convolution Layers tensorplay.nn.modules.conv.Conv1d tensorplay.nn.modules.conv.Conv2d tensorplay.nn.modules.conv.Conv3d tensorplay.nn.modules.conv.ConvTranspose1d tensorplay.nn.modules.conv.ConvTranspose2d tensorplay.nn.modules.conv.ConvTranspose3d tensorplay.nn.modules.conv.LazyConv1d tensorplay.nn.modules.conv.LazyConv2d tensorplay.nn.modules.conv.LazyConv3d tensorplay.nn.modules.conv.LazyConvTranspose1d tensorplay.nn.modules.conv.LazyConvTranspose2d tensorplay.nn.modules.conv.LazyConvTranspose3d tensorplay.nn.modules.folding.Unfold Extracts sliding local blocks from a batched input tensor (torch torch.nn.Unfold ). tensorplay.nn.modules.folding.Fold Combines an array of sliding local blocks into a large containing tensor (torch torch.nn.Fold ). Pooling layers tensorplay.nn.modules.pooling.MaxPool1d Applies a 1D max pooling over an input signal composed of several input planes. tensorplay.nn.modules.pooling.MaxPool2d Applies a 2D max pooling over an input signal composed of several input planes. tensorplay.nn.modules.pooling.MaxPool3d Applies a 3D max pooling over an input signal composed of several input planes. tensorplay.nn.modules.pooling.MaxUnpool1d Computes a partial inverse of MaxPool1d . tensorplay.nn.modules.pooling.MaxUnpool2d Computes a partial inverse of MaxPool2d . tensorplay.nn.modules.pooling.MaxUnpool3d Computes a partial inverse of MaxPool3d . tensorplay.nn.modules.pooling.AvgPool1d Applies a 1D average pooling over an input signal composed of several input planes. tensorplay.nn.modules.pooling.AvgPool2d Applies a 2D average pooling over an input signal composed of several input planes. tensorplay.nn.modules.pooling.AvgPool3d Applies a 3D average pooling over an input signal composed of several input planes. tensorplay.nn.modules.pooling.FractionalMaxPool2d Applies a 2D fractional max pooling over an input signal composed of several input planes. tensorplay.nn.modules.pooling.FractionalMaxPool3d Applies a 3D fractional max pooling over an input signal composed of several input planes. tensorplay.nn.modules.pooling.LPPool1d Applies a 1D power-average pooling over an input signal composed of several input planes. tensorplay.nn.modules.pooling.LPPool2d Applies a 2D power-average pooling over an input signal composed of several input planes. tensorplay.nn.modules.pooling.LPPool3d Applies a 3D power-average pooling over an input signal composed of several input planes. tensorplay.nn.modules.pooling.AdaptiveMaxPool1d Applies a 1D adaptive max pooling over an input signal composed of several input planes. tensorplay.nn.modules.pooling.AdaptiveMaxPool2d Applies a 2D adaptive max pooling over an input signal composed of several input planes. tensorplay.nn.modules.pooling.AdaptiveMaxPool3d Applies a 3D adaptive max pooling over an input signal composed of several input planes. tensorplay.nn.modules.pooling.AdaptiveAvgPool1d Applies a 1D adaptive average pooling over an input signal composed of several input planes. tensorplay.nn.modules.pooling.AdaptiveAvgPool2d Applies a 2D adaptive average pooling over an input signal composed of several input planes. tensorplay.nn.modules.pooling.AdaptiveAvgPool3d Applies a 3D adaptive average pooling over an input signal composed of several input planes. Padding Layers tensorplay.nn.modules.padding.ReflectionPad1d Pads the input tensor using the reflection of the input boundary. tensorplay.nn.modules.padding.ReflectionPad2d Pads the input tensor using the reflection of the input boundary. tensorplay.nn.modules.padding.ReflectionPad3d Pads the input tensor using the reflection of the input boundary. tensorplay.nn.modules.padding.ReplicationPad1d Pads the input tensor using replication of the input boundary. tensorplay.nn.modules.padding.ReplicationPad2d Pads the input tensor using replication of the input boundary. tensorplay.nn.modules.padding.ReplicationPad3d Pads the input tensor using replication of the input boundary. tensorplay.nn.modules.padding.ZeroPad1d Pads the input tensor boundaries with zero. tensorplay.nn.modules.padding.ZeroPad2d Pads the input tensor boundaries with zero. tensorplay.nn.modules.padding.ZeroPad3d Pads the input tensor boundaries with zero. tensorplay.nn.modules.padding.ConstantPad1d Pads the input tensor boundaries with a constant value. tensorplay.nn.modules.padding.ConstantPad2d Pads the input tensor boundaries with a constant value. tensorplay.nn.modules.padding.ConstantPad3d Pads the input tensor boundaries with a constant value. tensorplay.nn.modules.padding.CircularPad1d Pads the input tensor using circular padding of the input boundary. tensorplay.nn.modules.padding.CircularPad2d Pads the input tensor using circular padding of the input boundary. tensorplay.nn.modules.padding.CircularPad3d Pads the input tensor using circular padding of the input boundary. Non-linear Activations (weighted sum, nonlinearity) tensorplay.nn.modules.activation.ELU Applies elu: max(0, x) + min(0, alpha * (exp(x) - 1)) . tensorplay.nn.modules.activation.Hardshrink Applies the Hard Shrinkage (Hardshrink) function element-wise. tensorplay.nn.modules.activation.Hardsigmoid Applies hardsigmoid, element-wise: ReLU6(x + 3) / 6 . tensorplay.nn.modules.activation.Hardtanh Applies the HardTanh function element-wise. tensorplay.nn.modules.activation.Hardswish Applies hardswish, element-wise: x * ReLU6(x + 3) / 6 . tensorplay.nn.modules.activation.LeakyReLU Applies leaky_relu: max(0, x) + negative_slope * min(0, x) . tensorplay.nn.modules.activation.LogSigmoid Applies the Logsigmoid function element-wise. tensorplay.nn.modules.multihead_attention.MultiheadAttention Allows the model to jointly attend to information from different representation subspaces, as described in the paper Attention Is All You Need . tensorplay.nn.modules.activation.PReLU Applies the element-wise PReLU function. tensorplay.nn.modules.activation.ReLU Applies the rectified linear unit function element-wise. tensorplay.nn.modules.activation.ReLU6 Applies the element-wise function ReLU6(x) = min(max(0, x), 6) . tensorplay.nn.modules.activation.RReLU Applies the randomized leaky rectified linear unit function, element-wise. tensorplay.nn.modules.activation.SELU Applies selu with ATen's fixed lambda/alpha constants. tensorplay.nn.modules.activation.CELU Applies celu: max(0, x) + min(0, alpha * (exp(x / alpha) - 1)) . tensorplay.nn.modules.activation.Sigmoid Applies the Sigmoid function element-wise. tensorplay.nn.modules.activation.SiLU Applies the Sigmoid Linear Unit (SiLU) function, element-wise. tensorplay.nn.modules.activation.Mish Applies mish: x * tanh(softplus(x)) . tensorplay.nn.modules.activation.Softplus Applies softplus with linearization above threshold * beta . tensorplay.nn.modules.activation.Softshrink Applies the soft shrinkage function element-wise. tensorplay.nn.modules.activation.Softsign Applies the element-wise function: tensorplay.nn.modules.activation.Tanh Applies the Hyperbolic Tangent (Tanh) function element-wise. tensorplay.nn.modules.activation.Tanhshrink Applies element-wise, \(\text{Tanhshrink}(x) = x - \text{Tanh}(x)\) tensorplay.nn.modules.activation.Threshold Thresholds each element of the input Tensor. tensorplay.nn.modules.activation.GLU Applies the Gaussian Error Linear Units function. Non-linear Activations (other) tensorplay.nn.modules.activation.Softmin Applies the Softmin function to an n-dimensional input Tensor. tensorplay.nn.modules.activation.Softmax Softmax over dim , mirroring torch.nn.Softmax. tensorplay.nn.modules.activation.LogSoftmax Log-softmax over dim , mirroring torch.nn.LogSoftmax. tensorplay.nn.modules.adaptive.AdaptiveLogSoftmaxWithLoss Efficient softmax approximation. Normalization Layers tensorplay.nn.modules.batchnorm.BatchNorm1d Applies Batch Normalization over a 2D or 3D input. tensorplay.nn.modules.batchnorm.BatchNorm2d Applies Batch Normalization over a 4D input. tensorplay.nn.modules.batchnorm.BatchNorm3d Applies Batch Normalization over a 5D input. tensorplay.nn.modules.batchnorm.LazyBatchNorm1d A tensorplay.nn.BatchNorm1d module with lazy initialization. tensorplay.nn.modules.batchnorm.LazyBatchNorm2d A tensorplay.nn.BatchNorm2d module with lazy initialization. tensorplay.nn.modules.batchnorm.LazyBatchNorm3d A tensorplay.nn.BatchNorm3d module with lazy initialization. tensorplay.nn.modules.normalization.GroupNorm Applies Group Normalization over a mini-batch of inputs. tensorplay.nn.modules.batchnorm.SyncBatchNorm Applies Batch Normalization over a N-Dimensional input with synchronized batch statistics across all processes in the group. tensorplay.nn.modules.instancenorm.InstanceNorm1d Applies Instance Normalization. tensorplay.nn.modules.instancenorm.InstanceNorm2d Applies Instance Normalization. tensorplay.nn.modules.instancenorm.InstanceNorm3d Applies Instance Normalization. tensorplay.nn.modules.instancenorm.LazyInstanceNorm1d A tensorplay.nn.InstanceNorm1d module with lazy initialization of the num_features argument. tensorplay.nn.modules.instancenorm.LazyInstanceNorm2d A tensorplay.nn.InstanceNorm2d module with lazy initialization of the num_features argument. tensorplay.nn.modules.instancenorm.LazyInstanceNorm3d A tensorplay.nn.InstanceNorm3d module with lazy initialization of the num_features argument. tensorplay.nn.modules.normalization.LayerNorm Applies Layer Normalization over a mini-batch of inputs. tensorplay.nn.modules.normalization.LocalResponseNorm Applies local response normalization over an input signal. tensorplay.nn.modules.normalization.RMSNorm Applies Root Mean Square Layer Normalization over a mini-batch of inputs. Recurrent Layers tensorplay.nn.modules.rnn.RNNBase Base class for RNN modules (RNN, LSTM, GRU). tensorplay.nn.modules.rnn.RNN __init__(input_size, hidden_size, num_layers=1, nonlinearity='tanh', bias=True, batch_first=False, dropout=0.0, bidirectional=False, device=None, dtype=None) tensorplay.nn.modules.rnn.LSTM __init__(input_size, hidden_size, num_layers=1, bias=True, batch_first=False, dropout=0.0, bidirectional=False, proj_size=0, device=None, dtype=None) tensorplay.nn.modules.rnn.GRU __init__(input_size, hidden_size, num_layers=1, bias=True, batch_first=False, dropout=0.0, bidirectional=False, device=None, dtype=None) tensorplay.nn.modules.rnn.RNNCell An Elman RNN cell with tanh or ReLU non-linearity. tensorplay.nn.modules.rnn.LSTMCell A long short-term memory (LSTM) cell. tensorplay.nn.modules.rnn.GRUCell A gated recurrent unit (GRU) cell. Transformer Layers tensorplay.nn.modules.transformer.Transformer A basic transformer layer. tensorplay.nn.modules.transformer.TransformerEncoder TransformerEncoder is a stack of N encoder layers. tensorplay.nn.modules.transformer.TransformerDecoder TransformerDecoder is a stack of N decoder layers. tensorplay.nn.modules.transformer.TransformerEncoderLayer TransformerEncoderLayer is made up of self-attn and feedforward network. tensorplay.nn.modules.transformer.TransformerDecoderLayer TransformerDecoderLayer is made up of self-attn, multi-head-attn and feedforward network. Linear Layers tensorplay.nn.modules.linear.Identity A placeholder identity operator that is argument-insensitive. tensorplay.nn.modules.linear.Linear Applies an affine linear transformation to the incoming data: \(y = xA^T + b\) . tensorplay.nn.modules.linear.Bilinear Applies a bilinear transformation to the incoming data: \(y = x_1^T A x_2 + b\) . tensorplay.nn.modules.lazy.LazyLinear A tensorplay.nn.Linear module where in_features is inferred. Dropout Layers tensorplay.nn.modules.dropout.Dropout During training, randomly zeroes some of the elements of the input tensor with probability p . tensorplay.nn.modules.dropout.Dropout1d Randomly zero out entire channels. tensorplay.nn.modules.dropout.Dropout2d Randomly zero out entire channels. tensorplay.nn.modules.dropout.Dropout3d Randomly zero out entire channels. tensorplay.nn.modules.dropout.AlphaDropout Applies Alpha Dropout over the input. tensorplay.nn.modules.dropout.FeatureAlphaDropout Randomly masks out entire channels. Sparse Layers tensorplay.nn.modules.sparse.Embedding tensorplay.nn.modules.sparse.EmbeddingBag Distance Functions tensorplay.nn.modules.distance.CosineSimilarity Returns cosine similarity between \(x_1\) and \(x_2\) , computed along dim . tensorplay.nn.modules.distance.PairwiseDistance Computes the pairwise distance between input vectors, or between columns of input matrices. Loss Functions tensorplay.nn.modules.loss.L1Loss tensorplay.nn.modules.loss.MSELoss tensorplay.nn.modules.loss.CrossEntropyLoss tensorplay.nn.modules.loss.CTCLoss tensorplay.nn.modules.loss.NLLLoss tensorplay.nn.modules.loss.PoissonNLLLoss tensorplay.nn.modules.loss.GaussianNLLLoss tensorplay.nn.modules.loss.KLDivLoss tensorplay.nn.modules.loss.BCELoss tensorplay.nn.modules.loss.BCEWithLogitsLoss tensorplay.nn.modules.loss.MarginRankingLoss tensorplay.nn.modules.loss.HingeEmbeddingLoss tensorplay.nn.modules.loss.MultiLabelMarginLoss tensorplay.nn.modules.loss.HuberLoss tensorplay.nn.modules.loss.SmoothL1Loss tensorplay.nn.modules.loss.SoftMarginLoss tensorplay.nn.modules.loss.MultiLabelSoftMarginLoss tensorplay.nn.modules.loss.CosineEmbeddingLoss tensorplay.nn.modules.loss.MultiMarginLoss tensorplay.nn.modules.loss.TripletMarginLoss tensorplay.nn.modules.loss.TripletMarginWithDistanceLoss Vision Layers tensorplay.nn.modules.pixelshuffle.PixelShuffle Rearrange elements in a tensor according to an upscaling factor. tensorplay.nn.modules.pixelshuffle.PixelUnshuffle Reverse the PixelShuffle operation. tensorplay.nn.modules.upsampling.Upsample Upsamples a given multi-channel 1D (temporal), 2D (spatial) or 3D (volumetric) data. tensorplay.nn.modules.upsampling.UpsamplingNearest2d Applies a 2D nearest neighbor upsampling to an input signal composed of several input channels. tensorplay.nn.modules.upsampling.UpsamplingBilinear2d Applies a 2D bilinear upsampling to an input signal composed of several input channels. Shuffle Layers tensorplay.nn.modules.channelshuffle.ChannelShuffle Divides and rearranges the channels in a tensor. DataParallel Layers (multi-GPU, distributed) tensorplay.nn.parallel.data_parallel.DataParallel Implements data parallelism at the module level. tensorplay.nn.parallel.distributed.DistributedDataParallel Implements distributed data parallelism (torch parity). Utilities tensorplay.nn.utils.rnn.PackedSequence Holds the data and list of batch_sizes of a packed sequence. tensorplay.nn.utils.rnn.pack_padded_sequence Packs a Tensor containing padded sequences of variable length. tensorplay.nn.utils.rnn.pad_packed_sequence Pad a packed batch of variable length sequences. tensorplay.nn.utils.rnn.pad_sequence Pad a list of variable length Tensors with padding_value . tensorplay.nn.utils.rnn.pack_sequence Packs a list of variable length Tensors. tensorplay.nn.utils.rnn.unpack_sequence Unpack PackedSequence into a list of variable length Tensors. tensorplay.nn.utils.rnn.unpad_sequence Unpad padded Tensor into a list of variable length Tensors. tensorplay.nn.utils.rnn.invert_permutation Returns the inverse of permutation . tensorplay.nn.parameter.is_lazy Returns whether param is an UninitializedParameter or UninitializedBuffer . tensorplay.nn.factory_kwargs Return a canonicalized dict of factory kwargs. tensorplay.nn.modules.flatten.Flatten Flattens a contiguous range of dims into a tensor. tensorplay.nn.modules.flatten.Unflatten Unflattens a tensor dim expanding it to a desired shape. Lazy Modules Initialization tensorplay.nn.modules.lazy.LazyModuleMixin A mixin for modules that lazily initialize parameters, also known as "lazy modules". TensorPlay-specific additions Buffer A kind of Tensor that should not be considered a model parameter. DepthwiseConv2d NonDynamicallyQuantizableLinear Parameter A kind of Tensor that is to be considered a module parameter. RNNCellBase UninitializedBuffer A buffer that is not initialized. UninitializedParameter A parameter that is not initialized. # tensorplay.nn.init URL: https://www.tensorplay.cn/docs/nn.init.html tensorplay.nn.init torch.nn.init tensorplay.nn.init.calculate_gain tensorplay.nn.init.uniform_ tensorplay.nn.init.normal_ tensorplay.nn.init.constant_ tensorplay.nn.init.ones_ tensorplay.nn.init.zeros_ tensorplay.nn.init.eye_ tensorplay.nn.init.dirac_ tensorplay.nn.init.xavier_uniform_ tensorplay.nn.init.xavier_normal_ tensorplay.nn.init.kaiming_uniform_ tensorplay.nn.init.kaiming_normal_ tensorplay.nn.init.trunc_normal_ Fills the input Tensor with values drawn from a truncated normal distribution. tensorplay.nn.init.orthogonal_ tensorplay.nn.init.sparse_ # tensorplay.optim URL: https://www.tensorplay.cn/docs/optim.html tensorplay.optim Base class tensorplay.optim.optimizer.Optimizer Base class for optimizers. Module-level hooks tensorplay.optim.optimizer.register_optimizer_step_post_hook tensorplay.optim.optimizer.register_optimizer_step_pre_hook Utilities tensorplay.optim.swap_in_optimizer_params_and_state Temporarily install replacement parameters and packed optimizer state. Algorithms tensorplay.optim.adadelta.Adadelta Adadelta optimizer matching torch.optim.Adadelta . tensorplay.optim.Adafactor Implements Adafactor algorithm. tensorplay.optim.adagrad.Adagrad tensorplay.optim.adam.Adam tensorplay.optim.adamw.AdamW Adam with decoupled weight decay, matching torch.optim.AdamW . tensorplay.optim.sparse_adam.SparseAdam SparseAdam implements a masked version of the Adam algorithm suitable for sparse gradients. tensorplay.optim.adamax.Adamax Adamax optimizer matching torch.optim.Adamax . tensorplay.optim.asgd.ASGD Averaged stochastic gradient descent. tensorplay.optim.lbfgs.LBFGS Limited-memory BFGS optimizer, aligned with torch.optim.LBFGS . tensorplay.optim.Muon Implements Muon algorithm. tensorplay.optim.nadam.NAdam tensorplay.optim.radam.RAdam tensorplay.optim.rmsprop.RMSprop RMSprop optimizer matching Torch's centered and momentum variants. tensorplay.optim.rprop.Rprop Resilient backpropagation optimizer. tensorplay.optim.sgd.SGD How to adjust learning rate tensorplay.optim.lr_scheduler.LRScheduler Base class for all learning rate schedulers. tensorplay.optim.lr_scheduler.LambdaLR Sets the initial learning rate. tensorplay.optim.lr_scheduler.MultiplicativeLR Multiply the learning rate of each parameter group by the factor given in the specified function. tensorplay.optim.lr_scheduler.StepLR Decays the learning rate of each parameter group by gamma every step_size epochs. tensorplay.optim.lr_scheduler.MultiStepLR Decays the learning rate of each parameter group by gamma once the number of epoch reaches one of the milestones. tensorplay.optim.lr_scheduler.ConstantLR Multiply the learning rate of each parameter group by a small constant factor. tensorplay.optim.lr_scheduler.LinearLR Decays the learning rate of each parameter group by linearly changing small multiplicative factor. tensorplay.optim.lr_scheduler.ExponentialLR Decays the learning rate of each parameter group by gamma every epoch. tensorplay.optim.lr_scheduler.PolynomialLR Decays the learning rate of each parameter group using a polynomial function in the given total_iters. tensorplay.optim.lr_scheduler.CosineAnnealingLR Set the learning rate of each parameter group using a cosine annealing schedule. tensorplay.optim.lr_scheduler.ChainedScheduler Chains a list of learning rate schedulers. tensorplay.optim.lr_scheduler.SequentialLR Contains a list of schedulers expected to be called sequentially during the optimization process. tensorplay.optim.lr_scheduler.ReduceLROnPlateau Reduce learning rate when a metric has stopped improving. tensorplay.optim.lr_scheduler.CyclicLR Sets the learning rate of each parameter group according to cyclical learning rate policy (CLR). tensorplay.optim.lr_scheduler.OneCycleLR Sets the learning rate of each parameter group according to the 1cycle learning rate policy. tensorplay.optim.lr_scheduler.CosineAnnealingWarmRestarts Set the learning rate of each parameter group using a cosine annealing schedule. Putting it all together: EMA tensorplay.optim.swa_utils.AveragedModel A model that maintains a running SWA or EMA copy of another model. tensorplay.optim.swa_utils.SWALR Anneal each optimizer learning rate to a fixed SWA learning rate. tensorplay.optim.swa_utils.get_ema_avg_fn tensorplay.optim.swa_utils.get_swa_avg_fn tensorplay.optim.swa_utils.get_swa_multi_avg_fn tensorplay.optim.swa_utils.get_ema_multi_avg_fn tensorplay.optim.swa_utils.update_bn # Quantization URL: https://www.tensorplay.cn/docs/quantization.html Quantization Quantization API Reference (Kept since APIs are still public) # tensorplay.random URL: https://www.tensorplay.cn/docs/random.html tensorplay.random Random Generator tensorplay.manual_seed tensorplay.seed tensorplay.initial_seed tensorplay.get_rng_state tensorplay.set_rng_state tensorplay.random.fork_rng Forks the RNG state: code inside the context gets a pristine RNG. tensorplay.default_generator # tensorplay.sparse URL: https://www.tensorplay.cn/docs/sparse.html tensorplay.sparse Tensor methods and sparse tensorplay.to_sparse tensorplay.to_sparse_csr tensorplay.to_dense Torch functions specific to sparse Tensors tensorplay.functional.sparse_coo_tensor tensorplay.sparse.sum Sum of input 's values over dim . tensorplay.sparse.mm Performs a matrix multiplication of a 2-D sparse COO/CSR tensor with a dense matrix. tensorplay.sparse.spdiags Constructs a sparse tensor from diagonals, mirroring torch.sparse.spdiags (and thereby scipy.sparse.spdiags ). # tensorplay.special URL: https://www.tensorplay.cn/docs/special.html tensorplay.special Functions tensorplay.functional.airy_ai tensorplay.functional.bessel_j0 tensorplay.functional.bessel_j1 tensorplay.functional.bessel_y0 tensorplay.functional.bessel_y1 tensorplay.functional.chebyshev_polynomial_t tensorplay.functional.chebyshev_polynomial_u tensorplay.functional.chebyshev_polynomial_v tensorplay.functional.chebyshev_polynomial_w tensorplay.functional.digamma tensorplay.special.entr Entropy of a probability element: -x*log(x) (0 at x=0, NaN below). tensorplay.functional.erf tensorplay.functional.erfc tensorplay.special.erfcx Scaled complementary error function: exp(x^2) * erfc(x) . tensorplay.functional.erfinv tensorplay.functional.exp2 tensorplay.special.expit Numerically-stable sigmoid; identical to sigmoid() . tensorplay.functional.expm1 tensorplay.functional.gammainc tensorplay.functional.gammaincc tensorplay.special.gammaln Natural log of the absolute value of the gamma function (native lgamma). tensorplay.functional.hermite_polynomial_h tensorplay.functional.hermite_polynomial_he tensorplay.functional.i0 tensorplay.functional.i0e tensorplay.functional.i1 tensorplay.functional.i1e tensorplay.functional.laguerre_polynomial_l tensorplay.functional.legendre_polynomial_p tensorplay.functional.log1p tensorplay.special.log_ndtr log of the standard normal CDF, stable in the left tail. tensorplay.special.log_softmax tensorplay.functional.logit tensorplay.special.logsumexp Log-sum-exp along dim , computed with the max-shift trick. tensorplay.special.modified_bessel_i0 Modified Bessel function of the first kind, order 0; alias of i0() . tensorplay.functional.modified_bessel_i1 tensorplay.functional.modified_bessel_k0 tensorplay.functional.modified_bessel_k1 tensorplay.special.multigammaln Multivariate log-gamma with dimension p (p >= 2). tensorplay.special.ndtr Standard normal CDF: Phi(x) = 0.5 * erfc(-x / sqrt(2)) . tensorplay.special.ndtri Inverse standard normal CDF: sqrt(2) * erfinv(2x - 1) . tensorplay.special.polygamma Polygamma of order :attr:n: \(\psi^{(n)}(x)\) (native kernel). tensorplay.special.psi Digamma function; alias of digamma() . tensorplay.special.round Rounds to nearest even integer (native Tensor.round ). tensorplay.functional.scaled_modified_bessel_k0 tensorplay.functional.scaled_modified_bessel_k1 tensorplay.functional.shifted_chebyshev_polynomial_t tensorplay.functional.shifted_chebyshev_polynomial_u tensorplay.functional.shifted_chebyshev_polynomial_v tensorplay.functional.shifted_chebyshev_polynomial_w tensorplay.functional.sinc tensorplay.special.softmax tensorplay.functional.spherical_bessel_j0 tensorplay.special.xlog1py x*log1p(y) with the convention x*log1p(-1) = 0 when x == 0. tensorplay.special.xlogy x*log(y) with the convention 0*log(anything) = 0 . tensorplay.functional.zeta # tensorplay.stax URL: https://www.tensorplay.cn/docs/stax.html tensorplay.stax Static-graph optimization and acceleration. stax traces eager execution into a static graph, applies compiler passes (constant folding, dead code elimination, operator decomposition), and lowers the result to the available backends. This module is TensorPlay-specific and has no upstream torch counterpart. Functions stax Compile one canonical graph and return an executable callable. is_available Return whether the native Stax extension is loaded. # tensorplay URL: https://www.tensorplay.cn/docs/tensorplay.html tensorplay Tensors tensorplay.is_tensor Returns True if obj is a TensorPlay tensor. tensorplay.functional.is_complex tensorplay._composite_funcs.is_conj tensorplay._composite_funcs.is_neg tensorplay._composite_funcs.is_nonzero tensorplay._composite_funcs.is_same_size tensorplay.set_default_dtype Sets the default floating point dtype to d . tensorplay.get_default_dtype Returns the current default floating point dtype (float32 initially, changed by set_default_dtype() ). tensorplay.set_default_device Sets the default tensorplay.Tensor to be allocated on device . tensorplay.get_default_device Gets the default tensorplay.Tensor to be allocated on device tensorplay._composite_funcs.numel tensorplay.set_printoptions Set print options Creation Ops tensorplay.tensor tensorplay.functional.sparse_coo_tensor tensorplay.as_tensor Convert data into a tensor, sharing storage when possible. tensorplay.from_dlpack tensorplay.functional.zeros tensorplay.functional.zeros_like tensorplay.functional.ones tensorplay.functional.ones_like tensorplay.functional.arange tensorplay.functional.linspace tensorplay.functional.logspace tensorplay.functional.eye tensorplay.functional.empty tensorplay.functional.empty_like tensorplay.functional.full tensorplay.functional.full_like tensorplay.functional.quantize_per_tensor tensorplay.functional.quantize_per_channel tensorplay.functional.complex tensorplay.functional.polar tensorplay._composite_funcs.scalar_tensor tensorplay.functional.heaviside Indexing, Slicing, Joining, Mutating Ops tensorplay._composite_funcs.adjoint tensorplay._composite_funcs.alias_copy tensorplay.functional.argwhere tensorplay.functional.cat tensorplay._composite_funcs.concat tensorplay._composite_funcs.concatenate tensorplay.functional.conj tensorplay.functional.chunk tensorplay._composite_funcs.detach tensorplay._composite_funcs.diagonal_copy tensorplay.functional.dsplit tensorplay.functional.column_stack tensorplay.functional.dstack tensorplay._composite_funcs.expand_copy tensorplay.functional.fill tensorplay.functional.gather tensorplay.functional.hsplit tensorplay.functional.hstack tensorplay.functional.index_add tensorplay.functional.index_copy tensorplay.functional.index_put_ tensorplay.functional.index_select tensorplay.functional.masked_fill tensorplay.functional.masked_select tensorplay.functional.movedim tensorplay.functional.moveaxis tensorplay.functional.narrow tensorplay._composite_funcs.narrow_copy tensorplay.functional.nonzero tensorplay.functional.permute tensorplay._composite_funcs.permute_copy tensorplay._composite_funcs.put tensorplay.functional.reshape tensorplay.functional.row_stack tensorplay.functional.select tensorplay._composite_funcs.select_copy tensorplay.functional.scatter tensorplay.functional.diagonal_scatter tensorplay.functional.select_scatter tensorplay._composite_funcs.slice_copy tensorplay.functional.slice_scatter tensorplay.functional.scatter_add tensorplay.functional.split tensorplay._composite_funcs.split_copy tensorplay.functional.squeeze tensorplay._composite_funcs.squeeze_copy tensorplay.functional.stack tensorplay.functional.swapaxes tensorplay.functional.swapdims tensorplay.functional.t tensorplay._composite_funcs.t_copy tensorplay.functional.take tensorplay.functional.take_along_dim tensorplay.functional.tensor_split tensorplay.functional.tile tensorplay.functional.transpose tensorplay._composite_funcs.transpose_copy tensorplay.functional.unbind tensorplay._composite_funcs.unbind_copy tensorplay._composite_funcs.unfold_copy tensorplay._shape_funcs.unravel_index Converts flat indices into coordinate tuples (one LongTensor per dim). tensorplay.functional.unsqueeze tensorplay._composite_funcs.unsqueeze_copy tensorplay._composite_funcs.view_copy tensorplay.functional.vsplit tensorplay.functional.vstack tensorplay.functional.where Generators tensorplay.Generator Random sampling tensorplay.seed tensorplay.manual_seed tensorplay.initial_seed tensorplay.get_rng_state tensorplay.set_rng_state tensorplay.functional.bernoulli tensorplay.functional.multinomial tensorplay.functional.normal tensorplay.functional.poisson tensorplay.functional.rand tensorplay.functional.rand_like tensorplay.functional.randint tensorplay.functional.randint_like tensorplay.functional.randn tensorplay.functional.randn_like tensorplay.functional.randperm Serialization tensorplay.serialization.save Save obj to disk. tensorplay.serialization.load Load a checkpoint saved by tp.save or torch.save . Parallelism tensorplay.get_num_threads Returns the number of threads used for parallelizing CPU operations tensorplay.set_num_threads Sets the number of threads used for intraop parallelism on CPU. Locally disabling gradient computation tensorplay.autograd.grad_mode.no_grad Context-manager that disables gradient calculation. tensorplay.autograd.grad_mode.enable_grad Context-manager that enables gradient calculation. tensorplay.autograd.grad_mode.set_grad_enabled Context-manager that sets gradient calculation on or off. tensorplay.autograd.grad_mode.is_grad_enabled tensorplay.autograd.grad_mode.inference_mode Context manager that enables or disables inference mode. Pointwise Ops tensorplay.functional.abs tensorplay.functional.abs_ tensorplay._composite_funcs.absolute Alias of abs() . tensorplay.functional.acos tensorplay._composite_funcs.acos_ tensorplay._composite_funcs.arccos tensorplay.functional.acosh tensorplay._composite_funcs.acosh_ tensorplay._composite_funcs.arccosh tensorplay.functional.add tensorplay.functional.addcdiv tensorplay.functional.addcmul tensorplay.functional.angle tensorplay.functional.asin tensorplay._composite_funcs.asin_ tensorplay._composite_funcs.arcsin tensorplay.functional.asinh tensorplay._composite_funcs.asinh_ tensorplay._composite_funcs.arcsinh tensorplay.functional.atan tensorplay._composite_funcs.atan_ tensorplay._composite_funcs.arctan tensorplay.functional.atanh tensorplay._composite_funcs.atanh_ tensorplay._composite_funcs.arctanh tensorplay.functional.atan2 tensorplay._composite_funcs.arctan2 atan2(y, x) = 2*atan(y / (hypot + x)); native atan2 CPU kernel is currently unregistered in this tree, so compose. tensorplay.functional.bitwise_not tensorplay.functional.bitwise_and tensorplay.functional.bitwise_or tensorplay.functional.bitwise_xor tensorplay.functional.bitwise_left_shift tensorplay.functional.bitwise_right_shift tensorplay.functional.ceil tensorplay.functional.clamp tensorplay.functional.clamp_ tensorplay.functional.clip tensorplay._composite_funcs.copysign tensorplay.functional.cos tensorplay.functional.cosh tensorplay.functional.deg2rad tensorplay.functional.div tensorplay._composite_funcs.divide tensorplay.functional.digamma tensorplay.functional.erf tensorplay.functional.erfc tensorplay.functional.erfinv tensorplay.functional.exp tensorplay.functional.exp2 tensorplay.functional.expm1 tensorplay.functional.fill_ tensorplay.functional.fix tensorplay.functional.floor tensorplay._composite_funcs.floor_divide tensorplay._composite_funcs.fmod tensorplay.functional.frac tensorplay._composite_funcs.gradient tensorplay.functional.imag tensorplay.functional.lerp tensorplay.functional.lgamma tensorplay.functional.log tensorplay.functional.log10 tensorplay.functional.log1p tensorplay.functional.log2 tensorplay.functional.logaddexp tensorplay.functional.logaddexp2 tensorplay.functional.logical_and tensorplay.functional.logical_not tensorplay.functional.logical_or tensorplay.functional.logical_xor tensorplay.functional.logit tensorplay.functional.hypot tensorplay.functional.i0 tensorplay.functional.mul tensorplay._composite_funcs.multiply tensorplay.functional.nan_to_num tensorplay.functional.neg tensorplay.functional.neg_ tensorplay.functional.negative tensorplay.functional.nextafter tensorplay.functional.polygamma tensorplay.functional.positive tensorplay.functional.pow tensorplay.functional.rad2deg tensorplay.functional.real tensorplay.functional.reciprocal tensorplay._composite_funcs.remainder tensorplay.functional.round tensorplay.functional.rsqrt tensorplay.functional.rsqrt_ tensorplay.functional.sigmoid tensorplay.functional.sign tensorplay.functional.sgn tensorplay.functional.signbit tensorplay.functional.sin tensorplay.functional.sinc tensorplay.functional.sinh tensorplay.functional.softmax tensorplay.functional.sqrt tensorplay.functional.sqrt_ tensorplay.functional.square tensorplay.functional.sub tensorplay._composite_funcs.subtract tensorplay.functional.tan tensorplay.functional.tanh tensorplay._composite_funcs.true_divide tensorplay.functional.trunc tensorplay.functional.zero_ Reduction Ops tensorplay.functional.argmax tensorplay.functional.argmin tensorplay.functional.amax tensorplay.functional.amin tensorplay.functional.aminmax tensorplay.functional.all tensorplay.functional.any tensorplay.functional.max tensorplay.functional.min tensorplay.functional.dist tensorplay.functional.logsumexp tensorplay.functional.mean tensorplay.functional.nanmean tensorplay.functional.median tensorplay.functional.nanmedian tensorplay.functional.mode tensorplay.functional.norm tensorplay.functional.nansum tensorplay.functional.prod tensorplay._composite_funcs.quantile tensorplay._composite_funcs.nanquantile tensorplay.functional.std tensorplay.functional.std_mean tensorplay.functional.sum tensorplay.functional.unique tensorplay._composite_funcs.unique_consecutive tensorplay.functional.var tensorplay.functional.var_mean tensorplay.functional.count_nonzero Comparison Ops tensorplay.functional.allclose tensorplay.functional.argsort tensorplay.functional.eq tensorplay.functional.equal tensorplay.functional.ge tensorplay.functional.greater_equal tensorplay.functional.gt tensorplay.functional.greater tensorplay.functional.isclose tensorplay.functional.isfinite tensorplay._composite_funcs.isin tensorplay.functional.isinf tensorplay.functional.isposinf tensorplay.functional.isneginf tensorplay.functional.isnan tensorplay.functional.isreal tensorplay.functional.kthvalue tensorplay.functional.le tensorplay.functional.less_equal tensorplay.functional.lt tensorplay.functional.less tensorplay.functional.maximum tensorplay.functional.minimum tensorplay.functional.ne tensorplay.functional.not_equal tensorplay.functional.sort tensorplay.functional.topk tensorplay.functional.msort Spectral Ops tensorplay.functional.stft tensorplay.functional.istft tensorplay.functional.bartlett_window tensorplay.functional.blackman_window tensorplay.functional.hamming_window tensorplay.functional.hann_window tensorplay._composite_funcs.kaiser_window Other Operations tensorplay._composite_funcs.adaptive_avg_pool1d tensorplay._composite_funcs.adaptive_max_pool1d tensorplay.functional.atleast_1d tensorplay.functional.atleast_2d tensorplay.functional.atleast_3d tensorplay._composite_funcs.avg_pool1d tensorplay.functional.bincount tensorplay.functional.block_diag tensorplay.functional.broadcast_tensors tensorplay.functional.broadcast_to tensorplay._shape_funcs.broadcast_shapes Returns the broadcast shape of the given shapes (right-aligned). tensorplay.functional.bucketize tensorplay._composite_funcs.cartesian_prod tensorplay.functional.channel_shuffle tensorplay.functional.clone tensorplay._composite_funcs.combinations tensorplay.functional.conv1d tensorplay.functional.conv3d tensorplay.functional.conv_transpose1d tensorplay.functional.conv_transpose2d tensorplay.functional.conv_transpose3d tensorplay._composite_funcs.corrcoef tensorplay.functional.cosine_embedding_loss tensorplay._composite_funcs.cosine_similarity tensorplay._composite_funcs.cov Each row of input is a variable, each column an observation (torch semantics; no rowvar argument). tensorplay.functional.cummax tensorplay.functional.cummin tensorplay.functional.cumprod tensorplay.functional.cumsum tensorplay.functional.diag tensorplay.functional.diag_embed tensorplay._composite_funcs.diagflat tensorplay.functional.diagonal tensorplay.functional.diff tensorplay.functional.einsum tensorplay.functional.embedding tensorplay.functional.flatten tensorplay.functional.flip tensorplay.functional.gcd tensorplay.functional.group_norm tensorplay.functional.gru tensorplay.functional.hardshrink tensorplay.functional.hinge_embedding_loss tensorplay._composite_funcs.histc tensorplay._composite_funcs.histogram tensorplay.functional.instance_norm tensorplay.functional.kl_div tensorplay._composite_funcs.kron tensorplay.functional.lcm tensorplay.functional.logcumsumexp tensorplay.functional.lstm tensorplay._composite_funcs.lstm_cell tensorplay.functional.margin_ranking_loss tensorplay._composite_funcs.max_pool1d tensorplay.functional.meshgrid tensorplay.functional.pairwise_distance tensorplay.functional.pdist tensorplay.functional.pixel_unshuffle tensorplay.functional.poisson_nll_loss tensorplay.functional.prelu tensorplay.functional.ravel tensorplay.functional.relu_ tensorplay.functional.renorm tensorplay._composite_funcs.repeat_interleave tensorplay._composite_funcs.rms_norm tensorplay.functional.rnn_relu tensorplay._composite_funcs.rnn_relu_cell tensorplay.functional.rnn_tanh tensorplay._composite_funcs.rnn_tanh_cell tensorplay.functional.roll tensorplay.functional.rot90 tensorplay._composite_funcs.rsub tensorplay.functional.searchsorted tensorplay.functional.selu tensorplay._shape_funcs.tensordot Contracts input and other over the given dimensions. tensorplay.functional.threshold tensorplay.functional.trace tensorplay.functional.tril tensorplay._composite_funcs.tril_indices tensorplay.functional.triu tensorplay._composite_funcs.triu_indices tensorplay.functional.triplet_margin_loss tensorplay.functional.unflatten tensorplay._composite_funcs.vander tensorplay.functional.view_as_real tensorplay.functional.view_as_complex tensorplay._composite_funcs.resolve_conj tensorplay._composite_funcs.resolve_neg BLAS and LAPACK Operations tensorplay.functional.addbmm tensorplay.functional.addmm tensorplay.functional.addmv tensorplay.functional.addr tensorplay.functional.baddbmm tensorplay.functional.bmm tensorplay._composite_funcs.chain_matmul tensorplay.functional.cholesky_inverse tensorplay.functional.cholesky_solve tensorplay.functional.dot tensorplay._composite_funcs.ger tensorplay.functional.inner tensorplay.functional.matmul tensorplay._composite_funcs.matrix_power tensorplay.functional.mm tensorplay.functional.mv tensorplay.functional.outer tensorplay.functional.svd tensorplay._composite_funcs.trapz tensorplay._composite_funcs.trapezoid tensorplay._composite_funcs.cumulative_trapezoid tensorplay.functional.triangular_solve tensorplay.functional.vdot Utilities tensorplay.autocast_decrement_nesting tensorplay.autocast_increment_nesting tensorplay.clear_autocast_cache tensorplay.amp.autocast_mode.get_autocast_cpu_dtype Return the dtype to be used for CPU autocasting. tensorplay.get_autocast_dtype tensorplay.amp.autocast_mode.get_autocast_gpu_dtype Return the dtype to be used for CUDA autocasting. tensorplay._composite_funcs.get_device tensorplay.is_autocast_cache_enabled tensorplay.is_autocast_enabled tensorplay._composite_funcs.result_type tensorplay._composite_funcs.can_cast tensorplay._composite_funcs.promote_types tensorplay.set_autocast_cache_enabled tensorplay.set_autocast_dtype tensorplay.set_autocast_enabled tensorplay.use_deterministic_algorithms Sets whether TensorPlay operations must use "deterministic" algorithms. tensorplay.are_deterministic_algorithms_enabled Returns True if the global deterministic flag is turned on. tensorplay.is_deterministic_algorithms_warn_only_enabled Returns True if the global deterministic flag is set to warn only. tensorplay.set_deterministic_debug_mode Sets the debug mode for deterministic operations. tensorplay.get_deterministic_debug_mode Returns the current value of the debug mode for deterministic operations. tensorplay.set_float32_matmul_precision Sets the internal precision of float32 matrix multiplications. tensorplay.get_float32_matmul_precision Returns the current value of float32 matrix multiplication precision. tensorplay.typename String representation of the type of an object. Optimizations tensorplay.compiler.api.compile Compile a callable through the TensorPlay compiler frontend. TensorPlay-specific additions DType Members: Device DeviceType Members: GradScaler An instance scaler of GradScaler . Layout Sparse/dense storage layout tags (mirrors torch.layout loosely). MemoryFormat Tensor memory layout, mirroring torch.MemoryFormat. Scalar Size Tensor alias of TensorBase adaptive_avg_pool2d adaptive_avg_pool3d adaptive_max_pool2d add_ add_relu addcdiv_ addcmul_ airy_ai autocast Instances of autocast serve as context managers or decorators that allow regions of your script to run in mixed precision. avg_pool2d avg_pool3d batch_norm bernoulli_ bessel_j0 bessel_j1 bessel_y0 bessel_y1 binary_cross_entropy binary_cross_entropy_with_logits cauchy_ celu chebyshev_polynomial_t chebyshev_polynomial_u chebyshev_polynomial_v chebyshev_polynomial_w cholesky circular_pad_nd clamp_max clamp_min col2im constant_pad_nd contiguous conv1d_grad_bias conv1d_grad_input conv1d_grad_weight conv2d conv2d_grad_bias conv2d_grad_input conv2d_grad_weight conv2d_relu conv3d_grad_bias conv3d_grad_input conv3d_grad_weight conv_transpose1d_grad_bias conv_transpose1d_grad_input conv_transpose1d_grad_weight conv_transpose2d_grad_bias conv_transpose2d_grad_input conv_transpose2d_grad_weight conv_transpose3d_grad_bias conv_transpose3d_grad_input conv_transpose3d_grad_weight copy_ custom_bwd Create a helper decorator for backward methods of custom autograd functions. custom_fwd Create a helper decorator for forward methods of custom autograd functions. dequantize_per_channel dequantize_per_tensor device alias of Device div_ dtype alias of DType elu expand expand_as exponential_ fft_fft fft_ifft fft_irfft fft_rfft fork_rng Forks the RNG state: code inside the context gets a pristine RNG. forward_add forward_cos forward_div forward_exp forward_log forward_mm forward_mul forward_neg forward_pow forward_relu forward_sigmoid forward_sin forward_sqrt forward_sub forward_tanh fused_mul_add gammainc gammaincc gelu geometric_ get_parallel_info Returns a detailed string describing parallelization settings get_thread_num Returns the current thread number (starting from 0) in the current parallel region, or 0 in the sequential region glu hardsigmoid hardswish hardtanh hermite_polynomial_h hermite_polynomial_he huber_loss i0e i1 i1e im2col in_parallel_region Checks whether the code runs in a parallel region index_fill index_fill_ index_put inner_backward_other inner_backward_self inspect_checkpoint Summarize a checkpoint without loading tensor payloads. item l1_loss laguerre_polynomial_l layer_norm leaky_relu legendre_polynomial_p lerp_ log_normal_ log_softmax masked_fill_ masked_scatter matmul_backward_other matmul_backward_self max_pool2d mish modified_bessel_i1 modified_bessel_k0 modified_bessel_k1 mse_loss mul_ multi_margin_loss multilabel_margin_loss multilabel_soft_margin_loss native_dropout nll_loss normal_ one_hot pixel_shuffle quantized_linear random_ reflection_pad_nd relu relu6 repeat replication_pad_nd resize_ sample scaled_dot_product_attention scaled_modified_bessel_k0 scaled_modified_bessel_k1 scatter_ scatter_add_ shifted_chebyshev_polynomial_t shifted_chebyshev_polynomial_u shifted_chebyshev_polynomial_v shifted_chebyshev_polynomial_w silu slice smooth_l1_loss soft_margin_loss softplus softshrink sparse_add sparse_mm sparse_mul sparse_sum spdiags spherical_bessel_j0 split_with_sizes sub_ to_dense to_sparse to_sparse_csr tp_binary_cross_entropy tp_cosine_embedding_loss tp_hinge_embedding_loss tp_huber_loss tp_kl_div tp_l1_loss tp_margin_ranking_loss tp_poisson_nll_loss tp_smooth_l1_loss tp_soft_margin_loss unfold uniform_ unsafe_chunk unsafe_split upsample_bicubic2d upsample_bilinear2d upsample_linear1d upsample_nearest1d upsample_nearest2d upsample_nearest3d upsample_trilinear3d view zeta # Upstream anchor labels URL: https://www.tensorplay.cn/docs/upstream_labels.html Upstream anchor labels Reference pages keep torch docstrings verbatim, and some of them link to anchors that upstream defines in narrative sections this repo does not carry (see docs/README.md, “Deviations”). The labels below satisfy those links. Forward-mode AD Locally disabling gradient computation Reproducibility Backward-mode CUDA stream semantics CUDA semantics nn.init documentation # tensorplay.vision URL: https://www.tensorplay.cn/docs/vision.html tensorplay.vision tensorplay.vision is the computer-vision toolkit of TensorPlay, mirroring the torchvision package layout. The listing below is a static overview; the package requires optional dependencies (Pillow) for some entry points, so this page intentionally does not use autodoc. Top-level functions to_tensor , from_image , from_file — tensor/image conversion make_grid , save_image — visualization helpers set_backend , get_backend — image backend selection (PIL) Submodules tensorplay.vision.datasets — dataset classes and utilities ( MNIST , CIFAR10 / CIFAR100 , ImageFolder , Folder , DatasetFolder , UCF101 , …) tensorplay.vision.transforms — composable transforms ( Compose , ToTensor , Normalize , Resize , CenterCrop , RandomCrop , RandomHorizontalFlip , …) tensorplay.vision.models — reference model architectures ( AlexNet , VGG , ResNet variants, …) tensorplay.vision.io — image reading/writing backends tensorplay.vision.ops — vision-specific operators tensorplay.vision.utils — helpers shared across the package