Small Language Models

Small language models, often called SLMs, are compact versions of the large language models behind most AI assistants. Where frontier models may have hundreds of billions of parameters or more, small language models have a few billion or fewer. That makes them far cheaper and faster to run: they need less GPU memory, can serve many more requests per GPU, and in some cases can run on a laptop, phone or other device without any cloud connection. Many are released as open-weight models, while major providers also offer small models through their APIs at much lower prices than their flagship models. The quality of small models has improved quickly, and today's small models often outperform much larger models from only a few years ago.

Small models cannot match the breadth or reasoning ability of the largest models, but for many specific tasks they come very close, especially after fine-tuning or distillation. Classification, extraction, routing, simple question answering and summarization of short texts are common examples. For IT financial management, small language models are one of the most important tools for reducing AI costs, because a large share of everyday AI work does not need a frontier model, and paying frontier prices for it is a major source of waste.

Definition of Small Language Models

Small language models are language models designed to be efficient, with a parameter count small enough to run on modest hardware. There is no fixed threshold, but the term usually covers models from well under a billion to roughly ten billion parameters.

They are trained using the same methods as larger models, often with carefully selected data, and many are created by distilling knowledge from larger models, as described in model distillation.

Strengths

Small models are inexpensive per token, fast to respond and easy to host. Many can run on a single GPU, and some on CPUs or mobile devices.

They are well suited to high-volume, well-defined tasks, where their lower cost multiplies across large numbers of requests.

Running on local devices or private infrastructure, they can keep sensitive data within the organization and work without a network connection.

Limitations

Small models have less general knowledge and weaker reasoning than large models. They are more likely to make mistakes on complex, open-ended or unfamiliar tasks.

They often need fine-tuning or careful prompting to perform well on a specific task, and their performance can drop sharply on inputs unlike their training data.

For these reasons, small models are usually used alongside larger ones rather than replacing them entirely. Knowing where those limits lie for each task is what makes it safe to use them.

Importance of Small Language Models in IT Financial Management

Price differences between small and large models are large, often a factor of ten or more per token. Moving suitable work to small models is therefore one of the biggest single levers on AI spending.

Small models also lower infrastructure costs for self-hosted AI, because they need less memory and fewer or cheaper GPUs.

Their speed can improve user experience and make AI practical for real-time uses, such as autocomplete or voice, where larger models would be too slow. They also make it easier to run AI at the edge, close to where data is created.

Matching Models to Tasks

Most organizations have a mix of tasks, from simple to complex. Small models can handle the simple majority, while large models are reserved for the complex minority.

An LLM router can make this division automatically, sending each request to the cheapest model capable of handling it.

The resulting model mix, meaning the share of requests or tokens handled by small models, is a useful measure of how efficiently an organization uses AI.

Owning the Model

Open-weight small models can be fine-tuned and hosted by the organization, giving control over cost, data and behavior.

Hosting a small model is far simpler and cheaper than hosting a large one, which makes self-hosted LLM deployments practical for more organizations.

The trade-off is the effort of maintaining the model, monitoring its quality and updating it as tasks change. For many organizations, using a small model through a provider's API is a simpler first step before considering self-hosting.