Groups
Transformer expressiveness studies what kinds of sequence-to-sequence mappings a Transformer can represent or approximate.
Sinusoidal positional encoding represents each token’s position using pairs of sine and cosine waves at exponentially spaced frequencies.