On-device AI: why your next model may run on a laptop

On-device AI, sometimes called local AI or edge AI, means the model runs on the hardware in front of you. Your prompt, documents and the model's answer never leave the machine. A local LLM is the same idea applied to large language models: you download the model weights once and run them on a laptop, desktop or phone.

That is different from the way most people use AI today, where every request travels to a provider's data centre and is billed per token. Cloud AI is still the right choice for the hardest tasks. But a growing share of everyday work no longer needs a frontier model in a data centre.

Four developments made local models practical for business use.

Laptop chips with serious neural processors

Microsoft's Copilot+ PC standard requires a neural processing unit (NPU) capable of 40+ TOPS (trillion operations per second), along with 16 GB of RAM and 256 GB of storage. Qualifying chips include AMD Ryzen AI 300 and 400 series, Intel Core Ultra 200V and 300V series and Snapdragon X series processors. [1] In January 2026, Qualcomm announced the mainstream Snapdragon X2 Plus with a Hexagon NPU rated at 80 TOPS, with devices from leading manufacturers due in the first half of 2026. [2] Neural processors are moving from premium machines into ordinary business laptops.

Small open-weight models that run offline

Google released Gemma 4 on 2 April 2026 under the commercially permissive Apache 2.0 licence. Its two edge models, E2B and E4B, are built to "run completely offline with near-zero latency across edge devices like phones", feature native audio input for speech recognition and have a 128K token context window. Larger versions in the family target laptop GPUs. [3] Open weights mean a business can run the model on its own hardware without per-token fees.

Apple opened its on-device model to developers

At WWDC 2026, Apple said its on-device model is gaining vision capabilities and that the Foundation Models framework is going open source. Apple's server model, running on Private Cloud Compute, is available "with no cloud API costs to developers who have less than 2 million first time downloads". [4] Apple also introduced Core AI for running custom models on its devices, with "no server to manage, no cost per token, and no latency to the cloud". [5]

Microsoft made local models a supported product

Microsoft made Foundry Local generally available on 9 April 2026. It runs on Windows, macOS on Apple Silicon and Linux, picks the GPU, NPU or CPU automatically, caches models on the device, "works offline" and has "no per-token costs". Its curated catalogue includes GPT OSS, Qwen, Whisper, DeepSeek, Mistral and Phi models. [6] Running a local model on a company laptop is now a supported path, not a hobbyist project.

Small local models are not frontier models, and pretending otherwise leads to disappointment. They are very good at narrow, repetitive work where the input is in front of them:

• Checking and proofreading. Spelling, consistency and simple layout rules, the kind of always-on assistance that is uneconomic in the cloud.
• Transcription. Speech-to-text for meetings, field visits and voice notes, now built into small models such as Gemma 4's edge versions. [3] Our article on voice ordering for field sales shows why that matters for cost.
• Summarising and extracting. Pulling fields from invoices, forms and emails into structured data.
• Classifying and routing. Tagging tickets, documents or messages before a person or a larger model handles them.
• Working offline. Sites with poor connectivity, travel, and secure environments where internet access is restricted.

Complex reasoning, long multi-step analysis and the hardest coding tasks still favour large hosted models. The skill is knowing which is which.

It removes one big risk: the data never leaves the device, so there is no third-party processing to assess for that task. That helps with privacy obligations and with client confidentiality.

It does not remove every risk. A laptop running a local model still needs device encryption, managed updates and control over which models staff can install. Open-weight models should come from trusted publishers and be tested before use. Logging and oversight are harder when AI runs on hundreds of separate machines, so decide what must be recorded centrally.

For regulated organisations, local models sit alongside private AI on your own servers. Quantana runs open models on its own Nvidia-powered pods or on a client's existing infrastructure, and SecGPT Vault is our private AI that runs on-premises with no internet connection. Our private AI guide covers deployment options and APRA's expectations in detail.

Most organisations will use all three: device, private servers and cloud. A simple way to decide per use case:

• Run on the device when the task is narrow, frequent, latency-sensitive or must work offline, and when sending the data out is undesirable.
• Run on private servers when many users share the workload, the data is sensitive and you need central logging and control.
• Use the cloud when the task needs the strongest reasoning available, demand is occasional, or speed to launch matters more than unit cost.

Route between them. A small local model can handle the first pass and hand only the hard cases to a hosted model, which is one of the levers in our guide to reducing AI running costs. That guide also covers the break-even point between hosted and self-hosted models.

We continually benchmark new releases, and we choose each model per use case after evaluations for quality, speed, cost and integration effort. If you want to know which of your AI workloads could move off the cloud, get your AI roadmap.