On-Device AI: When Apple Intelligence Beats the Cloud (and When It Doesn't)

You shipped a feature with a fancy cloud AI behind it. Users loved it. Then the first invoice landed, because a pricing page that makes sense in the abstract makes none at all multiplied by everyone who uses your app.

The good news: you no longer have to put every AI call on a meter. Apple Intelligence and Google's Gemini Nano put genuinely capable models on the phone itself, where they run for free. The trick is knowing which calls belong there.

What on-device AI actually does well

"On-device" doesn't mean a crippled version of the big model. It means a real, small model tuned to be excellent at a handful of common, well-defined tasks — a sharp knife instead of a Swiss Army knife. Where it's genuinely good:

• Text classification. "Is this email spam, urgent, or routine?" Small models nail this.
• Summarizing one piece of content. One email, one note, one article. Short in, short out.
• Intent parsing. Turning "remind me to call mom tomorrow at 3" into structured data your app can act on.
• Simple rewrites. Tone, grammar, "make this more formal," "shorten this."
• Entity extraction. Names, dates, addresses, and amounts out of unstructured text.

If your feature lives here, on-device isn't just the budget choice. It's often the better product choice: instant, works on the plane, never sends private text to a third party.

What it can't do

The limits are real, and no amount of clever prompting removes them:

• Long-context reasoning. Drop a 200-page PDF on it and ask synthesis questions and you'll be disappointed.
• World knowledge and current events. The on-device model was frozen at training time and stays frozen until the OS updates it.
• Top-tier capability tasks. Production-quality code, nuanced long-form writing, genuinely hard multi-step reasoning. The gap has narrowed; it has not closed.

Treat it as a brilliant intern: plenty you can hand off, and work that still belongs to the senior engineer in the cloud.

How we actually got here

Most writing on this smears three separate releases into one vague "recently," so be precise:

• iOS 18.1, in late 2024, was the first Apple Intelligence developer surface: the Writing Tools APIs, alongside Genmoji and Image Playground. Writing Tools appeared automatically inside standard text controls, and the APIs mostly let you tune or opt out. Useful, but not a model you could prompt.
• The Foundation Models framework, announced at WWDC 2025 and shipped in iOS 26 in September 2025, was the real unlock: a first-class on-device LLM API for Swift, with guided generation for structured output and tool calling so the model can reach back into your app. No keys, no billing, no network.
• Gemini Nano on Android became the same story for the other half of the market, reached through ML Kit's GenAI APIs over AICore. Ask the OS, get inference, no bill.

The model Apple exposes on-device is roughly 3 billion parameters, runs on Apple Intelligence-capable hardware, and is free at inference time. The count matters less than what it implies: small, fast, tuned for exactly the summarize, classify, and extract work above.

What actually changed at WWDC 2026

This year's news isn't another on-device model. At WWDC 2026 Apple opened the Foundation Models framework to third-party LLM providers through a new LanguageModel protocol. Anything conforming to it, local or server-side, can back a LanguageModelSession — the same session type you already write against for the on-device model.

That turns the model behind a call into something close to a swappable part. Apple's session Bring an LLM provider to the Foundation Models framework lays out what can fill the slot:

• Apple's on-device model, rebuilt this year and now able to take images as input, not just text.
• A server-class model on Private Cloud Compute, with optional reasoning and no API keys or account setup to manage.
• Claude and Gemini, through Swift packages published by Anthropic and Google themselves rather than by Apple.
• Local open-weights models, run through Core AI or MLX on the Neural Engine and GPU.

The awkward part of a hybrid AI feature used to be the seam: on-device and cloud inference were different SDKs, different response shapes, different error handling, and the glue was yours to maintain. When both sides of an escalation speak the same session API, that seam mostly disappears. The routing decision is still yours; acting on it just got much cheaper. We covered the wider economics of that shift in How AI Pricing Really Works (and How Not to Blow Your Budget).

One caution: a unified API is not unified economics. A call routed to Claude or Gemini through a Foundation Models session is still metered on someone's bill, and in your code it looks identical to the free on-device one. Whatever decides where a call goes had better know what that call costs.

The router pattern

The pattern that wins is routing: every AI call passes through a small piece of logic that decides where it goes. Which cloud model sits at the far end of an escalation is its own decision, worked through in our honest, no-hype comparison of the 2026 models. The routing itself is simpler than it sounds. For each request, ask four questions:

1. Does this task need world knowledge or current events? If yes, cloud.
2. Does this task involve a lot of context — long documents, knowledge bases, deep histories? If yes, cloud.
3. Is the input sensitive? Medical, financial, personal messages. If yes, on-device by default, with explicit user consent before any cloud escalation.
4. Is the task simple, structured, or latency-sensitive? Classification, extraction, short summarization, intent parsing. On-device.

Four questions, decided in code, every request. Done well the user never knows which model answered — the app just feels fast and the bill stays flat. If you've never built one, our guide How to Build Your First AI Agent walks through the request-routing skeleton you'd reuse here.

Cost, latency, privacy: a three-way trade

No single call forces the choice, but across an app the axes add up. Cloud is per-token and tokens scale with users in ways founders consistently underestimate; on-device is free. Cloud round-trips are perceptible even on fast networks; local inference on short prompts feels instant. Cloud calls make you part of a vendor's compliance story; on-device data doesn't move unless you move it. On genuinely hard tasks cloud still wins, and pretending otherwise produces frustrated users.

Teams that get this right pick a default (usually on-device) and design observable escalation paths for when capability demands it. Teams that get it wrong default everything to the cloud and discover the bill the month they get a press hit.

Apple gave indie devs an AI runtime that costs nothing. The trick is knowing when "free" is good enough.

Designing for graceful fallback

Two realities break naive implementations. Apple Intelligence needs recent hardware, so older iPhones and iPads can't run the on-device models at all. And not every user is online, so "call the cloud when on-device isn't there" makes your app useless on the subway. Build a three-step ladder:

1. On-device, when available. Check at runtime, never at build time.
2. Cloud, when allowed and online. With user awareness if the data is sensitive.
3. A non-AI fallback when neither is available — a templated reply, a default classification, a "save and try again later." Boring, but unbreakable.

Apple's annual platform changes, covered in our indie-developer recap of WWDC 2026, keep expanding which devices count as "available." Build the ladder once and you ride that expansion for free.

Where to run common AI tasks

Use this as a default, then measure on your real data.

Where people go wrong (and when to call a pro)

The expensive mistakes are consistent enough to list:

If the bill matters or the data is sensitive, the value of hiring someone who has done this before isn't writing the prompt. It's making sure your routing logic doesn't quietly cost you a salary's worth of inference a month. That's where our app and AI engineering work usually starts.

Frequently asked questions