Scaling Laws
Empirical findings showing predictable relationships between model performance and factors like model size (parameters), dataset size, and compute budget. Performance improves as a power law with these factors.
Why It Matters
Scaling laws guide the multi-billion dollar decisions about how much to invest in training AI models. They predict performance before spending the compute.
Example
Doubling the model size from 7B to 14B parameters might reduce loss by ~15%, and this relationship holds predictably across many orders of magnitude.
Think of it like...
Like the economics of factory production — there are reliable rules about how output improves as you invest more in equipment, workers, and raw materials.
Related Terms
Parameter
Any learnable value in a machine learning model that is adjusted during training. Parameters include weights and biases in neural networks. Model size is often described by parameter count.
Frontier Model
The most capable and advanced AI models available at any given time, typically characterized by the highest performance across multiple benchmarks. These models push the boundaries of AI capabilities.
Compute
The computational resources (processing power, memory, time) required to train or run AI models. Compute is measured in FLOPs (floating-point operations) and is a primary constraint and cost in AI development.