Large language model

Large language models (LLMs) are AI systems that generate and interpret text by learning patterns from very large datasets. Prominent examples include GPT-3 and GPT-4, which helped popularize large-scale pretrained models and their downstream use, and ChatGPT, which reached a broad public audience through conversational access. ^[1][2][3]

Scaling the size of pretrained language models and the amount of training data has been shown to improve task-agnostic performance, including few-shot behavior where models follow instructions from a small number of examples. ^[1]

Research on compute-optimal training suggests that model size and training tokens should scale together to maximize performance under a fixed compute budget, informing how large models are trained and deployed. ^[4]

Architecture

Most contemporary LLMs are built on the Transformer architecture, which uses attention mechanisms to model relationships within sequences and supports large-scale sequence transduction. ^[5]

Capabilities

LLMs can perform a range of language tasks—such as translation, question answering, summarization, and code-related tasks—by conditioning on prompts rather than task-specific training data. Empirical results on large benchmarks show that scaling improves performance on many tasks and that instruction-following can be elicited with carefully constructed prompts. ^[1][2]

Limitations and risks

Despite strong performance, LLMs can produce confident but incorrect outputs, inherit biases from training data, and be misused for harmful or deceptive content. Training and deployment are resource-intensive, with environmental and economic costs, and large-scale web datasets can introduce documentation debt and amplify social biases when used without careful curation. The mechanisms behind emergent capabilities remain only partially understood. ^[6][7]