liveBook · Manning

Managed API

As explained in LLM Customization and Fine-Tuning

A provider-operated model interface accessed through an API rather than self-hosted by the user.

A managed API is an externally hosted model accessed through an HTTP endpoint. Also called model-as-a-service or hosted inference, it lets a team send prompts and pay per token while the provider handles infrastructure, updates, reliability, and scaling. section 2.4.1

The decision is whether faster delivery and lower operational burden outweigh vendor dependence and the need to send data to the provider. Vendor model versions, deprecations, pricing, and terms remain management concerns even when infrastructure maintenance is outsourced. section 2.4.1 section 2.4

Start with a managed API when:

  • The problem is still being explored: prompt iteration can reveal capability gaps quickly without a training run. section 2.4.1
  • A hosted model can solve the task without fine-tuning, as with summarization, translation, code generation, or questions over provided context. section 2.4.1
  • The team wants to avoid managing GPU clusters, CUDA configuration, deployments, and latency monitoring. section 2.4.1
Listing 2.1 Calling a managed API (Anthropic Python SDK)
import anthropic
 
client = anthropic.Anthropic()
message = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=1000,
    messages=[
        {
            "role": "user",
            "content": "Explain the difference between a stock and a bond "
                       "in two sentences.",
        }
    ],
)
print(message.content[0].text)

This Anthropic Python SDK call sends a prompt to a hosted model. It requires the SDK and an ANTHROPIC_API_KEY environment variable; other providers use their own clients, credentials, and model identifiers. section 2.4.1

Reconsider self-hosting when hard privacy or residency requirements prohibit sending data to a vendor, deeper customization or weight access is necessary, or traffic makes owned infrastructure cheaper beyond a break-even point. Self-hosting also brings an ongoing serving and maintenance commitment. section 2.4.1 section 2.11 section 2.4

Table 2.3 Buy versus build: criteria for choosing between a managed API and a self-hosted model.
Criterion
Buy: managed API
Build: self-hosted
Setup time
Minutes
Days to weeks to provision GPUs and configure the serving stack
Upfront cost
Low, pay per token
High, physical GPU hardware or cloud GPU VMs
Cost at scale
Grows linearly with usage
One-time infrastructure cost amortized across requests
Customization
Prompt engineering plus limited fine-tuning APIs
Fine-tuning, weight access, custom serving stack
Data privacy
Sent to vendor servers
Stays on your infrastructure
Model choice
Vendor decides versions and deprecations
Any open-weight model you choose
Latency control
Vendor SLA on shared infrastructure
You own the hardware and networking path
Maintenance burden
Zero infrastructure burden; vendor management overhead (deprecations, pricing, terms) is real.
Ongoing operational commitment for hardware, serving stack, and scaling

Compare setup time, cost, customization, privacy, latency control, and maintenance across managed and self-hosted models. section 2.4

How LLM Customization and Fine-Tuning covers it

How deep it goes
Explains selection criteria, operational responsibilities, and an SDK call, stopping short of a detailed vendor selection procedure.
Examples
Uses an Anthropic Python SDK call that asks a hosted model to distinguish stocks from bonds.
What you can do after
Evaluate whether a managed API or self-hosting better fits privacy, customization, traffic, and operational constraints.
How it connects
Connects managed APIs to prompting, fine-tuning, self-hosting, data privacy, and cost at scale.

Also explained in

Compare how 2 books explain managed API →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.