Suppose an NBFC's credit team has a prototype that reads a loan application and drafts the credit note. It was built in a week against a hosted model API, because that was the quickest way to get a draft on screen. Then the compliance head asks where the applicant data goes, and nobody has a written answer. Where the model runs fixes where your data goes, who carries the operations work and how hard the choice is to reverse, so it should be decided deliberately rather than inherited from the prototype.
What each option asks you to run
There are two broad options. A hosted model is one you call through a provider's API. An open-weight model is one whose trained parameters (the weights) are published, so you can run it on your own machines.
With a hosted model, your application sends a request over the internet to the provider and receives the response. You run nothing except your own code. The provider runs the model, the hardware, the scaling, the upgrades and the on-call rota.
With an open-weight model, you download the weights, run an inference server on GPU machines in your cloud account, and your application calls that server. You now own a serving stack: model files, serving software, machines, autoscaling, monitoring, upgrades, and whoever gets paged when it falls over at two in the morning.
- Provider runs model and hardware
- Requests leave your network
- Shared capacity, provider-managed
- Updates on the provider's schedule
- Pay per request
- You run the serving stack
- Requests stay in your account
- Capacity you provision
- Updates when you choose
- Pay for machines, busy or idle
Neither column is better in general. Each fits a specific set of constraints, and the rest of this post is about identifying yours.
Where the data goes
For most organisations this decides the matter. A hosted request carries the prompt, with whatever customer data your application put into it, to a third party. Providers publish terms on retention and on whether request data is used for training, and enterprise agreements can restrict both. The data still crosses a boundary, and your legal and security people will want to read those terms line by line. For the NBFC, that prompt contains PAN numbers, bank statement extracts and home addresses.
In your own cloud, the prompt never leaves your account. Data residency rules, sector regulation and customer contracts that forbid third-party processing are satisfied by the architecture itself. The remaining risk is your own: access control on the serving machines and on the logs, which hold exactly the same data the prompts did.
There is a middle path. Several cloud platforms host provider models inside a region you choose, under contractual data boundaries. That satisfies some policies and not others. Check with the people who own the policy before assuming it does, and get the answer in writing.
Latency and capacity
A hosted API adds a network round trip and shares capacity with every other customer, so response time varies with the provider's load, and rate limits apply. For an interactive feature, such as a credit officer waiting a few seconds for a draft note, that is usually acceptable. For a batch job that reprocesses a lakh of applications overnight, the rate limit can become the bottleneck, and the failure looks like a job still running at nine the next morning.
Self-hosted serving gives predictable latency inside your network and capacity that is yours alone, provided you provisioned enough. Traffic above that level queues or fails, and GPU machines take minutes to start, so you have to plan capacity instead of assuming it. A month-end spike in applications is exactly where this shows.
The operations cost nobody adds up
This is where cost comparisons usually go wrong. The hosted bill arrives as an invoice. The self-hosted cost includes GPU machines, which cost the same whether busy or idle, plus the engineering time to keep the serving stack patched, upgraded, monitored and available at night. Teams without an existing platform group tend to underestimate that second part by a wide margin, and the people doing the work are usually the ones who were meant to be building features.
Which option costs less depends on volume. Low, bursty traffic is almost always cheaper hosted, because you pay nothing while idle. High, steady traffic is where owned machines start to pay for themselves, and where per-request charges grow with every new branch that starts using the feature. Whichever you pick, put per-user budgets, caching and routing in place before the feature scales.
Updates and switching later
Hosted providers release new models, retire old ones, and occasionally change behaviour under the same model name. Your prompts and evaluation sets then have to be rerun on the provider's schedule. With open weights, nothing changes until you change it. That is reassuring for a regulated feature, and a burden when a much better model appears and you have to qualify it yourself, on your own GPUs, against your own test set.
- Your applicationcalls the serving endpoint
- Serving softwareloads weights, batches requests
- Model weightsa specific tagged release
- GPU machinesprovisioned, patched, scaled
- Monitoring and on-calllatency, errors, capacity
Each layer is something the hosted provider ran for you.
Lock-in runs in both directions. A hosted API ties you to its request format, its tool-calling conventions and prompts tuned to its model. Open weights tie you to a serving stack and a hardware commitment, and your prompts are just as tuned to one model. The way out is the same in both cases: keep prompts in version control, keep evaluation sets that run against any model, and put a thin adapter between your application and the model API. A switch then becomes a configuration change and a test run, not a rewrite.
A decision rule by sensitivity and volume
| Data sensitivity | Low or bursty volume | High, steady volume |
|---|---|---|
| Public or low-risk data | Hosted API | Hosted first; revisit on cost |
| Confidential, under contract limits | Hosted, with region and retention terms, if policy allows | Open-weight in your cloud |
| Regulated or residency-bound | Open-weight in your cloud, or no feature | Open-weight in your cloud |
Read the row first. If the data cannot leave, volume does not matter, and the NBFC's choice is made for it. If the data can leave, volume decides.
Where the comparison goes wrong
The most common mistake is choosing self-hosting for privacy and then sending the same data to a hosted provider for embeddings or for evaluation, which undoes the point. Next is comparing the hosted invoice with the GPU bill and leaving out the engineers. Then building the whole feature against one provider's SDK, so that by the time anyone runs the comparison, switching is no longer realistic.
Two more. Picking an open-weight model because it is free to download, without running it on your own evaluation set first; weights that fail your test set are not cheap at any price. And provisioning for peak traffic, then paying for idle machines most of the week.
Settle the data question first, with the people who own the policy, and write the answer down. It removes half the options before any cost estimate is needed. Then run your evaluation set on one candidate from each column before committing, and decide who will watch the feature once it is live; the minimum monitoring for a model in production applies to either choice.
