tensorplay.nn.attention.bias.causal_lower_right

tensorplay.nn.attention.bias.causal_lower_right(*size) → CausalBias[source]

Creates a lower-right triangular causal bias.

This function generates a lower-right triangular matrix to represent causal attention bias with a diagonal offset set so that the inclusive values are aligned to the lower right corner of the matrix.

The equivalent code for constructing this bias is:

diagonal_offset = size[1] - size[0]
tensorplay.tril(
    tensorplay.ones(size, dtype=tensorplay.bool),
    diagonal=diagonal_offset,
)

For instance, with shape=(3,4), the materialized bias tensor will be:

[[1, 1, 0, 0],
 [1, 1, 1, 0],
 [1, 1, 1, 1]]

Parameters:

size – The size of the bias matrix.

Returns:

The LOWER_RIGHT triangular causal bias variant.

Return type:

CausalBias