When to Use Local AI vs Frontier Models
The question usually arrives as a choice: local AI vs frontier models, rent or own, the system in your own building or the one you reach over the internet. Framed that way, it has no single good answer, because each option gives up something the other makes essential.
The better question is not which to use but which work runs where. Frontier models and local models are not competitors fighting for the same job; they are tiers, and the decision that matters is where you draw the line between them.
The Binary Is False
Almost every serious deployment ends up using both, because the two are good at different things and priced on entirely different logic. Posing it as an either/or is what makes the budget and the risk both look worse than they are.
A frontier model is the most capable reasoning engine you can rent by the token. A local model is capability you own outright and can point directly at your most sensitive data.
Once you stop treating those as rivals, the real work begins: deciding which of your tasks belongs to which tier.
What Frontier Models Buy
Frontier models buy raw reasoning, the kind that only comes out of training runs and hardware almost no one outside a handful of labs can field. A single frontier training run now costs somewhere in the tens to hundreds of millions of dollars, and that spending shows up as capability you could never reproduce in your own rack.1
For genuinely hard work, novel synthesis, long chains of inference, ambiguous problems with no template, that horsepower is usually worth the call. You are renting the output of enormous capital for the price of a few thousand tokens.
This is the honest case for the cloud, and it does not go away. The mistake is assuming every task needs that much engine.
What Only Local Promises
Local models deliver the one thing a frontier API structurally cannot: the data never leaves. Enterprise cloud contracts now offer zero-retention and no-training terms, and the reputable ones are written in good faith, but a contract is a promise and a network boundary is a fact.
Control beats trust. When the model runs on hardware you own and can audit end to end, the question of who else could see a client's files, an unsigned deal, or the formula that is your business has exactly one answer, and it is no one.
This is not distrust of any particular vendor. It is declining to extend trust at all for the work that would hurt most if it leaked.
Regulation Follows the Boundary
Most compliance questions are really questions about where data goes, so the boundary you draw answers them before the lawyers do. The GDPR turns on where personal data is processed and transferred, HIPAA requires a signed business-associate agreement with every vendor that touches protected health information,2 and the EU AI Act adds obligations by risk tier on top of all of it.3
When ingestion, retrieval, and inference all run on hardware you hold, the list of outside parties to vet collapses toward zero, and a long compliance questionnaire turns into one verifiable fact.
Send the same data to a frontier API and every one of those obligations returns, not because the cloud is careless, but because you have added a party to the transaction.
Context Windows and Token Budgets
A context window is how much the model can hold in mind at once: the running length of the conversation plus everything you have fed it. Frontier models now offer very large windows, up to a million tokens in at least one case, which is why they can reason across an entire case file or codebase.4
But every token in that window is billed and processed, so a big window is a budget, not a free resource. The practical skill is putting less in front of the model, not more.
This is where local work pays for itself twice. Routine preparation done on your own hardware, cleaning, extracting, summarizing, shrinks what you finally send up to the frontier model, and the bill shrinks with it.
Speed Is a Feeling
Throughput, measured in tokens per second, is what makes a chat or an agent feel responsive instead of sluggish. It is the difference between an assistant that keeps pace with you and one you sit and wait on.
For interactive work, a fast local model often feels better than a more capable remote one, because the round trip to a distant data center is latency the user pays on every single turn.
Speed and capability are not the same axis. The fastest answer and the best answer are frequently different models, and a good system routes to whichever the moment actually needs.
Small Models, Layered Right
Raw reasoning still favors the frontier, but well-structured small models do far more than their size suggests when they are layered correctly. The leverage is in the architecture, not in heroics from any single model.
Spread the work across subagents, each handling one slice of the problem in its own context window, and a coordinating model can stay focused on the decision instead of drowning in detail.5
The orchestrator keeps its brainpower for the call that matters while cheaper models do the legwork. A team of small models with clear roles routinely outperforms one large model asked to hold everything at once.
Retrieval Focuses the Frontier
The highest-leverage way to spend less on a frontier model is to send it only what matters, which is the entire job of a retrieval system. Hand the model the right three paragraphs and it answers well on a fraction of the tokens.
Good retrieval is also the hardest part to build, a stack of keyword search, vector search, fusion, and reranking that each cover a different failure mode. Systems like FactoryOS treat retrieval, not generation, as the decisive engineering precisely because that is where answers are won or lost.
Done well it is invisible: the answers are simply right and the bill is simply lower. Done poorly, it quietly hands the model the wrong page and you pay full price for a confident mistake.
Drawing Your Line
The line is not fixed, but the logic for drawing it is. Send work to the frontier when the task is genuinely hard and the data is not sensitive; keep work local when it is routine, high-volume, or confidential, which describes most of what an organization actually does all day.
Done this way the two tiers subsidize each other. Local hardware absorbs the steady, sensitive bulk, the frontier model is reserved for the few calls that earn it, and the total cost of ownership lands well below either extreme.
The real question for a buying committee was never local vs cloud AI. It is which of your work is too sensitive, too repetitive, or too costly to send away, and the answer to that is an architecture, not a subscription.