tensorplay.nn.attention.bias.causal_upper_left

tensorplay.nn.attention.bias.causal_upper_left(*size) → CausalBias[source]

Creates an upper-left triangular causal bias.

This function generates an upper-left triangular matrix to represent causal attention bias with a diagonal offset set so that the inclusive values are aligned to the upper left corner of the matrix. This equivalent to the is_causal=True argument in scaled_dot_product_attention.

The equivalent code for constructing this bias is:

tensorplay.tril(tensorplay.ones(size, dtype=tensorplay.bool))

For instance, with shape=(3,4), the materialized bias tensor will be:

[[1, 0, 0, 0],
 [1, 1, 0, 0],
 [1, 1, 1, 0]]

Parameters:

size – The size of the bias matrix.

Returns:

The UPPER_LEFT triangular causal bias variant.

Return type:

CausalBias