Large language model

Large language models (LLMs) are neural network language models trained on very large text corpora to predict and generate natural language. Contemporary LLMs are typically built on the Transformer architecture, which replaces recurrent or convolutional sequence models with attention mechanisms for sequence transduction. Prominent examples include GPT-3 and GPT-4, which helped popularize large-scale pretrained models and their downstream use, and ChatGPT, which reached a broad public audience through conversational access. ^[1][2][3][4]

Scaling the size of pretrained language models and the amount of training data has been shown to improve task-agnostic performance, including few-shot behavior where models follow instructions from a small number of examples. ^[2]

Research on compute-optimal training suggests that model size and training tokens should scale together to maximize performance under a fixed compute budget, informing how large models are trained and deployed. ^[5]

Capabilities

LLMs can perform a range of language tasks—such as translation, question answering, summarization, and code-related tasks—by conditioning on prompts rather than task-specific training data. Empirical results on large benchmarks show that scaling improves performance on many tasks and that instruction-following can be elicited with carefully constructed prompts. ^[2][3]

Limitations and risks

Despite strong performance, LLMs can produce confident but incorrect outputs, inherit biases from training data, and be misused for harmful or deceptive content. They are also expensive to train and run, raising environmental and economic concerns, and their emergent capabilities are not yet fully understood. ^[6]

Critical analyses have emphasized the environmental costs of training large models, the risks of documentation debt in massive web-scale datasets, and the amplification of social biases and harms when uncurated data are used at scale. ^[7]