A small language model (SLM) is a scaled-down language model with fewer parameters than a large language model. It can generate text, summarize documents, translate languages, and answer questions while requiring less compute to train and operate. That makes it attractive when an enterprise needs lower costs, faster responses, or deployment on a device with limited processing capacity. section 2.6 section 2.9
Choose an SLM for the task, not just for its size:
- Compare quality on the intended workload. SLMs generally trail larger models on complex tasks requiring broad context, although some match or outperform larger models on particular tasks. section 2.6
- Consider where it will run. Lower compute requirements open up local and edge deployment, but capacity, reliability, and scaling still need planning. section 2.6 section 11.1
- Keep model choice flexible. A model layer can expose specialized SLMs and general-purpose LLMs through a consistent interface, whether they are hosted internally or externally. section 10.2.3
Use a small model as an intent classifier
A question-answering application can run Phi-2 locally as a first-pass classifier. A prompt asks whether a question concerns dogs; the classifier returns a yes-or-no decision that determines whether to call the more expensive GPT-4 model. The illustrative Python implementation loads Phi-2 with transformers and torch, tokenizes the question, and generates the classification. Treat this as a routing demonstration, not a production-ready classifier: test its decisions before relying on them to control access to a larger model. section 10.5.1
Listing 10.6 Using Phi-2 as an intent classifier
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
import openai
...
model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2",
torch_dtype="auto",
trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2",
trust_remote_code=True)
def check_dog_question(question):
prompt = f"Instruct: Is there anything about dogs in the
question below? If yes, answer with 'yes' else
'no'.\nQuestion:{question}\nOutput: "
inputs = tokenizer(prompt, return_tensors="pt",
return_attention_mask=False,
add_special_tokens=False)
outputs = model.generate(**inputs,
max_length=500,
pad_token_id=tokenizer.eos_token_id)
text = tokenizer.batch_decode(outputs)[0]
regex = "^Output: Yes$"
match = re.search(regex, text, re.MULTILINE)
if match:
return True
return False
def handle_dog_question(question):
print( "This is a response from RAG and GPT4")
# Call OpenAI's GPT-4 to answer the question
openai.api_key = "YOUR_API_KEY"
response = openai.Completion.create(
…
)
return response
if __name__=="__main__":
# Loop until the user enters "quit"
while True:
# Take user input
user_prompt = input(
"What is your question (or type 'quit' to exit):")
if check_dog_question(user_prompt):
print(handle_dog_question(user_prompt))
else:
print("You did not ask about dogs")
The Phi-2 listing shows the local model load, classification prompt, tokenization, and generation steps behind the first-pass decision. section 10.5.1
Phi-2, Phi-3, and Orca 2 illustrate different small-model sizes and capabilities. Smaller is not automatically better: evaluate task quality against compute and operating cost, and account for security, privacy, monitoring, and compliance when deploying an SLM alongside other models. section 10.5 section 11.1
