What are an AI model's "parameters"?
Parameters are the weights a model adjusts through training, and their count is a common measure of model size. It is usually written as a number, such as "70B (70 billion)."
You can think of more parameters as a larger "capacity" to hold more complex patterns.
Are more parameters always better?
Not necessarily. More parameters give more expressive power, but training and running the model need more compute and memory, and costs rise.
Depending on training data quality, architecture, and alignment (tuning) methods, a smaller model can even beat a larger one on specific tasks. Parameter count is best treated as one of several metrics.
What is MoE (Mixture of Experts)?
MoE (Mixture of Experts) is an architecture that puts multiple "expert" neural networks inside a model and activates only some of them for each input. Even if the total parameter count is large, only part of it is used at a time, so compute cost can be reduced.
This makes it possible to build models with a large total parameter count but light actual computation, so it is widely used in recent large models.
How do MoE models differ from regular (dense) models?
A regular dense model uses all of its parameters for every input. MoE instead selects and uses only the experts that fit the input, so it can have a larger total capacity for the same amount of computation.
The trade-off is that it needs "routing" to decide which experts to pick, and training and serving become more complex.