A model ensemble combines multiple machine-learning models so an application can use their complementary strengths, improve reliability, and handle varied user requests. section 10.5.1
Design an effective ensemble around these production concerns:
- Select models by performance, training-data diversity, or specialization. section 10.5.1
- Use routing logic to choose a model or combine multiple outputs. section 10.5.1
- Manage endpoint integration, throttling, rate limits, errors, and response caching. section 10.5.1
- Plan for scaling, load balancing, regional redundancy, queues, and stream processing. section 10.5.1
Intent routing for dog questions
Run Phi-2 locally as a fast classifier to determine whether a question concerns dogs; route irrelevant questions to GPT-4 for answering. section 10.5.1
Listing 10.6 Using Phi-2 as an intent classifier
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
import openai
...
model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2",
torch_dtype="auto",
trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2",
trust_remote_code=True)
def check_dog_question(question):
prompt = f"Instruct: Is there anything about dogs in the
question below? If yes, answer with 'yes' else
'no'.\nQuestion:{question}\nOutput: "
inputs = tokenizer(prompt, return_tensors="pt",
return_attention_mask=False,
add_special_tokens=False)
outputs = model.generate(**inputs,
max_length=500,
pad_token_id=tokenizer.eos_token_id)
text = tokenizer.batch_decode(outputs)[0]
regex = "^Output: Yes$"
match = re.search(regex, text, re.MULTILINE)
if match:
return True
return False
def handle_dog_question(question):
print( "This is a response from RAG and GPT4")
# Call OpenAI's GPT-4 to answer the question
openai.api_key = "YOUR_API_KEY"
response = openai.Completion.create(
…
)
return response
if __name__=="__main__":
# Loop until the user enters "quit"
while True:
# Take user input
user_prompt = input(
"What is your question (or type 'quit' to exit):")
if check_dog_question(user_prompt):
print(handle_dog_question(user_prompt))
else:
print("You did not ask about dogs")
Inspect a Python implementation that loads Phi-2, prompts it for a yes-or-no intent decision, and supports downstream routing. section 10.5.1
