Inside P10: Dispatcher and hardware abstraction

First define the boundary: Tensor and TensorImpl

p10::Tensor is the lightweight handle used by user code and operators. It holds a p10::TensorImpl smart pointer, allowing callers to pass tensors without carrying every runtime detail at each call site.

TensorImpl owns the runtime state: shape, stride, dtype, device, storage, and lifetime. CPUImpl, CUDAImpl, and EdgeImpl can provide different concrete implementations without changing how Tensor is passed through the API.

• Tensor: a stable, lightweight entry point for calls.
• TensorImpl: describes where data lives and how it is laid out and owned.
• Backend implementations: provide device-specific memory and kernel behavior.

Dispatcher: route an operator to the right kernel

Calls such as add() and matmul() do not hard-code CPU, CUDA, or an edge device at the call site. Dispatcher builds a DispatchKey from the input devices, dtypes, and operator signature, then selects a matching registered kernel.

This separates what the program wants to do from how a particular device does it. A new backend can focus on storage, registration, and kernels without copying the higher-level tensor API.

p10::Tensor y = p10::matmul(a, b)
  -> read a.device(), a.dtype(), b.device(), b.dtype()
  -> build DispatchKey
  -> look up the registered kernel
  -> execute on the selected backend

From API to hardware: a readable call path

Dispatch is not a black box that must be guessed from the final output. When reading an operator call, inspect the input TensorImpl, the fields that form the DispatchKey, the function selected from the registry, and the memory layout read by the kernel.

That makes debugging more local. A wrong result can be narrowed to shape or stride metadata, an incorrect key, an incomplete backend implementation, or the numerical behavior of one kernel.

Why backends can evolve independently

Once device abstraction, operator registration, and concrete implementations are separated, adding hardware mainly affects the backend boundary. Higher-level modules can keep using the same tensor operators, and TPX can continue to reuse its differentiation rules.

The advantage is not a claim that every device will be faster. It is that hardware experiments become easier to compare: run the same inputs, operators, and output contract on multiple backends, then inspect kernels, memory behavior, and numerical error.

• Stable interface: higher layers do not need each device’s storage details.
• Replaceable implementation: backend kernels can be registered, tested, and iterated independently.
• Verifiable result: device differences can be reduced to output, performance, and error comparisons.

How to verify Dispatcher instead of only checking output

When validating dispatch, observe both numerical results and the selected path. Comparing only the final tensor can miss a mistaken default device or an unexpectedly slow kernel.

• Fix the input shape, dtype, and device, then record the DispatchKey.
• Run the same operator on CPU and CUDA and compare error and boundary behavior.
• Use non-contiguous tensors or different strides to cover layout branches.
• Add operator-level tests for a new kernel instead of relying only on network-level regressions.