On-Device AI Models in 2026: Free Per Call, Not Free to Build For

Your app needs a language model for one narrow job: summarize a note, extract a date, rewrite a sentence in a friendlier tone. The default move in 2026 is still an API key, a few cents per call, and a feature that stops working the second the phone loses signal. On-device AI models are Apple's and Google's answer to that default: a language model already sitting on the phone, callable through a platform SDK, at zero marginal cost per call, that never sends a byte off the device, with a capability ceiling and a fragmented, platform-specific toolchain the marketing rarely mentions.

What does "on-device AI" actually mean in 2026?#

On-device AI in 2026 means a language model built directly into the phone's operating system, callable through a platform SDK, running on the phone's own chip instead of a data center. Apple ships this through its Foundation Models framework, Google through Gemini Nano and AICore on Android, both free to call and offline by default.

Both companies frame it the same way: the model ships with the OS, updates with the OS, and runs on-chip using the phone's own neural accelerator instead of a round trip to a server. Apple's version is a Swift framework; Google's is a set of Kotlin and Java APIs backed by a system service called AICore. Neither is a library you bundle into your app. Both are calls into something the OS already runs.

The numbers that anchor the rest of this piece, from Apple's own research posts and Android's developer docs.

The chip underneath both stacks matters more than either company advertises. Apple's own silicon carries the Neural Engine; on Android, inference usually runs on a Qualcomm Hexagon or MediaTek NPU, the dedicated accelerator block that makes running a model on battery power feasible at all. A Snapdragon 8 Elite's Hexagon cores, for one example, have been benchmarked running an 8-billion-parameter model at around 5 tokens per second, entirely on-chip. That NPU layer is the real enabling hardware. The OS-level SDK is just the door you knock on to reach it.

What can Apple's on-device Foundation Models framework actually do?#

Apple's Foundation Models framework, launched with iOS 26 and rebuilt for its third generation at WWDC 2026, gives developers a free, private, offline Swift API for a 3-billion-parameter on-device model, plus a larger 20-billion-parameter sparse model, both tuned for narrow tasks, not general knowledge.

The framework's core object is LanguageModelSession, and the appeal from the first release was how little ceremony it needed: no API key, no network permission, structured output defined with a @Generable annotation instead of a hand-rolled JSON schema. That shipped in iOS 26 in 2025.

WWDC 2026 split the lineup in two: AFM 3 Core, the next version of the original 3-billion-parameter dense model, and AFM 3 Core Advanced, a 20-billion-parameter model with a sparse architecture that activates only 1 to 4 billion parameters per request. The full model lives in flash storage rather than requiring it all fit in RAM, the trick that makes a 20-billion-parameter model plausible on a phone at all. The same update added image input, so a session can reason over a photo alongside text, entirely on-device.

The bigger 2026 change is not a bigger model, it is a protocol. Apple opened the framework to any LLM provider: implement a LanguageModel and a LanguageModelExecutor, and the same LanguageModelSession code can target Apple's on-device model, its cloud tier (Private Cloud Compute), a locally-packaged Hugging Face model, or a third-party API, without a rewrite. It does not raise the ceiling on the free on-device model, but outgrowing that ceiling no longer means throwing away the integration.

What does Google's Gemini Nano do on Android?#

Gemini Nano is Google's on-device model for Android, reached through ML Kit GenAI APIs and the AICore system service. The 2026 tier, marketed as Gemini Intelligence, needs Nano v3, a flagship chip, and 12GB of RAM, a stricter floor than Apple's 8GB Apple Intelligence baseline.

AICore is the system service that holds Gemini Nano on the device, manages updates, and isolates its processing from the network the same way Android's Private Compute Core isolates other sensitive work: data goes in, an answer comes out, nothing reaches Google in between. ML Kit's GenAI APIs sit on top: summarization, proofreading, rewriting, image description, plus a general Prompt API for custom text or multimodal input.

AICore itself launched Pixel-only and now runs on non-Pixel Android 14 and newer devices too, widening its reach considerably. The catch is the newest tier. Google's Gemini Intelligence branding, introduced at I/O 2026, sits on top of Gemini Nano v3 and gates it hard: 12GB of RAM minimum, a flagship-class chip, and a commitment to 5 or more years of OS updates. Multiple outlets reported that requirement locking out phones as recent as the Pixel 9 and Galaxy S25, both released with less RAM than the new floor demands. That is a stricter bar than Apple ever set for Apple Intelligence, which shipped at 8GB of RAM on the iPhone 15 Pro in 2024.

Read the table as two separate product lines, not a spectrum. No version of either model runs on the other company's OS, and no shared API surface connects them.

Why is zero inference cost the real headline, not raw model quality?#

The real headline for on-device AI is not model quality, it's the bill. A call to the on-device model carries no metered API charge and no per-token line item, because there is no API call, just a function call into the OS. No other layer of the AI stack works this way.

Every other tier of AI economics, a frontier model, a mid-tier API, even a self-hosted open-weight model on your own server, still has a real per-request or per-hour cost somewhere. Our breakdown of the local LLM versus API break-even point walks through that math for hardware you buy and run yourself, a fixed monthly cost that only pays off once volume clears a threshold. On-device AI skips that calculation entirely: the hardware is not yours to amortize, it is already in your user's pocket, paid for by them, running on power they already budget for.

The two are not competing on the same axis: one trades cost and privacy for a ceiling, the other trades cost for range.

That changes the shape of the cost curve completely. An API bill scales with usage, linearly, forever. An on-device feature's marginal cost per call is $0 for every user, whether you have 100 users or 10 million, because each phone runs its own inference. The tradeoff is that you also give up what an API gives you: a model you chose, at a version you control, that improves when the vendor improves it, not only when the user updates their OS.

What's the actual capability ceiling on a phone-sized model?#

Both companies deliberately cap the on-device model's size, which caps what it can do. Apple says its on-device model is built for language understanding, structured output, and tool calling, not general knowledge. Ask it something a browser tab would need to answer and it will guess, confidently, and wrongly.

That is not a bug to route around, it is the design. A 3-billion or even 20-billion-parameter model, sparse or dense, will never hold the breadth of a frontier model trained on a far larger share of the internet, and neither company is trying to make it. Google's own API list is the tell: summarize, proofread, rewrite, describe an image, transcribe speech, each a narrow, well-specified transformation of input the app already has in hand, not an open-ended question about the world.

Neither Apple nor Google publishes exactly how AFM 3 Core or Gemini Nano get compressed down to a phone-sized footprint, but the industry's general playbook for shrinking a frontier model's behavior into a small one is model distillation: train a small student model on a larger teacher's outputs instead of shipping the giant model itself. Assume something in that family runs upstream of every on-device release, even without a published paper for either company's specific pipeline.

The honest tradeoff: real savings and privacy against a capped model and double the integration work.

Long context is the other ceiling worth naming directly. Apple's own materials describe the on-device model as having no long-context mode worth relying on. If your task needs to reason over a long document or a long conversation history, that is a job for the cloud tier, not the free one.

Why can't you just build "for on-device" generically?#

There is no cross-platform on-device SDK. Apple's framework is Swift-only, built for iOS, iPadOS, and macOS. Google's is Kotlin and Java, built for Android through AICore. Different languages, different models, different RAM floors, different capabilities. Supporting both platforms on-device means writing and testing two separate integrations, not one.

This is the part the survey articles skip. "On-device AI" reads like one target you build toward, the way "the cloud" is one target for an API call. It is not. Apple's LanguageModelSession and Google's ML Kit GenAI Prompt API do not accept the same input shape, do not return the same structured-output format, and are not backed by the same model. WWDC 2026's provider protocol makes it easier to swap backends within Apple's own ecosystem, but it does nothing for the Apple-to-Android gap: still 2 SDKs, 2 languages, 2 RAM floors, and 2 sets of release notes to track.

If what you actually want is to choose your own model and run it on hardware you fully control, desktop or server-side, that is a different and more flexible trade, covered in our comparison of Ollama, LM Studio, and Jan. The platform SDKs in this piece trade that choice away in exchange for zero setup on the user's device and zero distribution cost to you.

Should this feature run on-device or call the cloud?#

There is a real decision here, not a default. Run the 3 gates in order: is the task narrow enough for a phone-sized model, is privacy or offline a hard requirement or is volume high enough to matter, and do your target devices actually clear the platform's RAM and chip floor.

Three gates decide it: the task's shape, whether privacy or volume justifies it, and whether target devices actually clear the floor.

Start with the task, not the platform. If it needs broad world knowledge, a long document, or reasoning outside what the app hands the model as context, that is a cloud job regardless of anything else here, skip straight there. If the task is narrow, classification, extraction, summarization, a rewrite, keep going.

Next, ask why you want on-device at all. A hard privacy or offline requirement justifies it on its own, no cost math needed. Short of that, on-device only pays off once call volume is high enough that a metered API bill would actually hurt, the same logic the cost break-even math runs for self-hosted hardware, just pushed further out to hardware you never had to buy.

Last, check the floor for real. Google's 12GB-plus-flagship-chip requirement for Gemini Intelligence and Apple's roughly 8GB Apple Intelligence baseline both exclude a meaningful share of phones still in active use. If your actual user base skews toward older or budget hardware, building on-device for only a fraction of them is a maintenance cost with a small payoff. Route everyone else to the cloud.

What should a small team actually do this week?#

Pick 1 narrow feature already on your roadmap, summarizing a note, tagging a category, rewriting a tone, and prototype it against the platform's free on-device model this week. That single afternoon tells you more about the real capability ceiling than any spec sheet, benchmark, or marketing page will.

A single afternoon spent prototyping answers more than another comparison article.

Build the smallest possible version first: 1 platform, 1 task, 1 real device that clears the RAM floor, no fallback yet. You are testing whether the on-device model is good enough for this specific job, not building the production feature. That answer usually arrives faster than a week.

If it holds up, add the cloud fallback for devices below the floor and ship it as a genuinely free feature: no API key to provision, no per-call cost to model, no outage to plan around. If it does not hold up, you have lost an afternoon, not a quarter, and you already know to route that task straight to the cloud model you would have used anyway.