Choosing an open-source model means running the model on infrastructure you control instead of relying on a hosted API. Its weights can be obtained and run under permitted terms, and may be adapted or fine-tuned. This can support privacy, lifecycle control, and domain-specific behavior, but transfers infrastructure and operational work to your team. section 2.1.2 section 10.2
Production ownership brings both control and responsibilities:
- Control where inference runs, how weights and model versions are managed, and how prompts, parameters, routing, and monitoring are configured. section 10.2.2
- Plan for GPUs, memory, serving software, load balancing, caching, quantization, monitoring, and demand-based scaling; operating these components requires engineering time. section 10.2.2
- Use control over training data, evaluation, and weights to detect and reduce bias, while accepting responsibility for those choices. section 11.3.1
A local Mistral deployment
A Mistral 7B model can run in a private environment and be fine-tuned on domain data. A well-tuned deployment may perform comparably to GPT-3.5 for some use cases, but requires hosting and maintenance rather than API-only integration. section 10.2.2 section 2.1.2
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
class LocalModelService:
def __init__(self, model_name="mistralai/Mistral-7B-Instruct-v0.1"):
self.tokenizer = AutoTokenizer.from_pretrained(model_name)
self.model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype=torch.float16,
device_map="auto"
)
def generate_response(self, prompt: str) -> str:
inputs = self.tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
outputs = self.model.generate(
inputs.input_ids,
max_new_tokens=500,
temperature=0.7,
do_sample=True,
pad_token_id=self.tokenizer.eos_token_id
)
response = self.tokenizer.decode(outputs[0], skip_special_tokens=True)
return response[len(prompt):].strip()
Python example loading a local model and tokenizer with Transformers, then generating a response with PyTorch. section 10.2.2
