Neural Networks
Positional Encoding
Neural Networks· Advanced
Definition
A mechanism that injects information about the position of each token in a sequence into a transformer model, since the attention mechanism itself is permutation-invariant. Without positional encodings, transformers cannot distinguish 'dog bites man' from 'man bites dog'.
Enterprise Context
Understanding positional encoding explains why transformers have context length limits and why different encoding schemes (RoPE, ALiBi) enable different long-context behaviors.
Tags
#transformer#architecture#attention
MS
Maxx Stacks Editorial
Reviewed by enterprise AI practitioners
Maxx University
Keep learning. Keep building.
250+ terms. 5 learning paths. AI maturity assessment. Jargon translator. All free, always.