Phi-2 is Microsoft's 2.7-billion-parameter small language model, designed to provide useful language processing with lower computational requirements than larger models. It can serve as an efficient first-stage model in an enterprise AI architecture. section 10.5
Use Phi-2 where a fast, lightweight decision can reduce reliance on a more expensive model:
- Run Phi-2 locally as an intent classifier before routing suitable requests to GPT-4. section 10.5.1
- Prompt the model to return a constrained answer such as “yes” or “no” when checking whether a question concerns a target topic. section 10.5.1
- Treat the model as less capable than a larger model, while using its speed and efficiency to control routing costs and latency. section 10.5 section 10.5.1
Topic routing with Phi-2
A local function can tokenize a question, prompt Phi-2 to determine whether it concerns dogs, and generate a Boolean-style result. A positive result can trigger GPT-4 for a detailed answer; a negative result avoids the more expensive call. section 10.5.1
Listing 10.6 Using Phi-2 as an intent classifier
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
import openai
...
model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2",
torch_dtype="auto",
trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2",
trust_remote_code=True)
def check_dog_question(question):
prompt = f"Instruct: Is there anything about dogs in the
question below? If yes, answer with 'yes' else
'no'.\nQuestion:{question}\nOutput: "
inputs = tokenizer(prompt, return_tensors="pt",
return_attention_mask=False,
add_special_tokens=False)
outputs = model.generate(**inputs,
max_length=500,
pad_token_id=tokenizer.eos_token_id)
text = tokenizer.batch_decode(outputs)[0]
regex = "^Output: Yes$"
match = re.search(regex, text, re.MULTILINE)
if match:
return True
return False
def handle_dog_question(question):
print( "This is a response from RAG and GPT4")
# Call OpenAI's GPT-4 to answer the question
openai.api_key = "YOUR_API_KEY"
response = openai.Completion.create(
…
)
return response
if __name__=="__main__":
# Loop until the user enters "quit"
while True:
# Take user input
user_prompt = input(
"What is your question (or type 'quit' to exit):")
if check_dog_question(user_prompt):
print(handle_dog_question(user_prompt))
else:
print("You did not ask about dogs")
Inspect the Python implementation showing model loading, tokenization, prompting, and generation with Phi-2. section 10.5.1
