Model selection chooses a predictive model that fits the task and its practical constraints. In finance, compare candidates on relevant real-world performance as well as cost, privacy, and operational needs. section 3.3.1 section 10.5
For generative AI in finance, weigh candidate models against the product’s needs:
- Commercial models can accelerate prototyping and offer strong out-of-the-box performance, but API use raises data-privacy, vendor-lock-in, and unpredictable-cost concerns. section 3.3.1
- Open-source models allow private-cloud or on-premise hosting, data sovereignty, and customization, at the cost of greater infrastructure investment and in-house expertise. section 3.3.1
- A task-specific comparison can reveal that a lower-cost candidate captures most of the meaning even when another produces more eloquent summaries. section 3.3.1
A two-track decision
Use a commercial API to launch an initial MVP and gather user feedback, while evaluating and testing open-source alternatives in parallel for longer-term use. section 3.3.1
Listing 3.1 A simple, practical litmus test for model evaluation
import openai
from typing import List
import os
# This snippet demonstrates a practical test to compare different models
# on a core task, going beyond generic benchmarks.
client = openai.OpenAI() # Assumes OPENAI_API_KEY is set in the environment
def run_model_test(model_name: str, post: str) -> str:
"""Sends a single, domain-specific post to a model for a quality check."""
prompt = f"""
You are a financial analyst. Summarize the key investment thesis from this post
in one clear sentence.
Post: "{post}"
Summary:
"""
try:
response = client.responses.create(
model=model_name,
input=prompt,
temperature=0.1,
max_output_tokens=100
)
return response.output_text.strip()
except openai.APIError as e:
return f"API error with model {model_name}: {e}"
# --- Example Test Case ---
test_post = "I'm still bullish on $TECHCO. Their recent acquisition in the AI space is a game-changer. PT $150."
# A list of candidate models to compare. You can update this list with
# the latest models available at the time of running.
models_to_test = ["gpt-4o", "gpt-4o-mini"]
print(f"--- Running Comparative Test for Post ---\n'{test_post}'\n")
for model in models_to_test:
print(f"--- Testing Model: {model} ---")
summary = run_model_test(model, test_post)
print(f"Response: {summary}\n")
# In a real scenario, you would also add open-source models to this list
# and run them using a library like Hugging Face's transformers.
Compare candidate models by asking each to summarize the investment thesis in a financial post, then inspect the outputs for task-specific quality. section 3.3.1
