A statistical model is autoregressive if it predicts future values based on past values. For example, an autoregressive model might seek to predict a stock’s future prices based on its past performance.
In context of LLMs, generative pre-trained transformers (GPTs) are derivations of auto-regressive models where it takes an input sequence of tokens length and predicting the next token at index .
Auto-regressive models are often considered a more correct terminology when describing text-generation models.
Important
The correct terminology when people refers to LLMs are transformers being trained for auto-regressive objectives
Not to be confused with encoder-decoder models (the original transformers papers propose encoder-decoder architecture, but this is mainly useful for translation)