What is an on-device or local LLM?
An on-device or local LLM is a language model that runs directly on the user's own device, such as a personal PC or smartphone, rather than on a cloud server. The key point is that data is processed inside the device without being sent out.
"On-device" mostly refers to models built into devices like smartphones and appliances, while "local LLM" refers to running a model directly on a personal PC or server, but the idea is the same.
How is it different from a cloud LLM?
A cloud LLM runs a large model on powerful servers, but your data is sent externally. A local LLM keeps data on the device, which is better for privacy, works without internet, and has no per-use fees.
The trade-off is that device limits make it hard to run very large models, so it becomes a matter of balancing performance against privacy and cost.
What hardware do you need to run a local LLM?
The requirements depend on model size. In general, the more parameters a model has, the more memory it needs (especially GPU VRAM or system RAM), and "quantization," which shrinks the model, lets it run on lower-spec hardware.
So choosing a model size that fits your device's specs is the first step to running locally.
How do you get started with a local LLM?
These days there are local runner tools that download and run models for you, so you can start with a few commands. A sensible approach is to confirm things work with a small model first, then scale up as far as your device can handle.
Choosing a specialized model for your use, such as chat or code, can give better results even on the same hardware.