I Ran a 4.5 Billion Parameter AI Model on a Phone. No Cloud. No API Keys. Here's How.

Last week, I asked a 4.5 billion parameter AI model "What's the weather in Tokyo?" — and it called a live weather API, parsed the response, and gave me a formatted answer. All running on a phone. No cloud. No API key. No data leaving the device. This isn't a Google I/O demo. It's an open-source Flutter app I built called EdgeAI Kit.

Why On-Device AI Matters

Every time your app sends user data to OpenAI or Google's API, you're taking on risk:

Privacy regulations — HIPAA, GDPR, and your enterprise clients always asking "where does our data go?"
API costs — Per-token pricing adds up fast at scale
Latency — Round-trip to a server adds 200-500ms minimum
Availability — No internet? No AI. Unless it runs locally.

On-device AI eliminates all of that. The model runs on hardware your users already own. Think about the use cases:

Healthcare apps processing patient notes without HIPAA cloud headaches
Finance apps analyzing documents without data ever leaving the device
Enterprise tools that work in airplane mode, in secure facilities, offline
Zero per-token API costs — forever

What I Built

EdgeAI Kit is a Flutter toolkit that makes on-device AI practical, not just possible. Here's what it does:

Model Catalog with Device-Aware Recommendations

The app scans your device hardware and tells you which models will run well. Gemma 4 E2B (2.4 GB) gets a "Great fit for your device" badge. Gemma 4 E4B (4.3 GB) gets a "Should work, may be slower" warning. No guessing.

Model catalog showing Gemma 4 variants with capability badges

Seven models are ready to download: Google Gemma 4, DeepSeek, Qwen, Phi-4, and more. Each shows its capabilities — Vision, Audio, Tools, Thinking — so you know exactly what you're getting.

On-Device Chat with Streaming

Once a model is downloaded, you can chat with it. Tokens stream in real-time as the model generates them — entirely on-device. I tested this by asking the model "Are you making an HTTP request?" and it confirmed: no network calls, no external access, pure local inference.

On-device chat with Gemma 4 showing streaming responses

Function Calling — On a Phone

This is the part that surprised me most. The AI model can call real tools. I defined three tools — get_current_time, get_weather, and calculate — and the model decides when to use them. Asked "What's the weather in Tokyo?" and Gemma 4 autonomously called the weather tool, received the JSON response, and formatted a human-readable answer. All on-device except the actual weather API call.

Function calling — AI model calling get_weather tool on device

This opens up serious possibilities: on-device AI agents that can interact with local databases, device sensors, or selective APIs — while keeping the reasoning entirely private.

Benchmark: Real Numbers

No hand-waving about performance. The built-in benchmark runs a standard prompt and measures actual throughput:

Benchmark results: 13 tokens/sec, 475ms time to first token

13.0 tokens/second on a mid-range phone
475ms time to first token
250 tokens generated in 18.6 seconds

Running Gemma 4 E4B — Google's latest open model that rivals GPT-4 class performance at a fraction of the size. Not blazing fast, but absolutely usable for real applications.

Prompt Lab

Full control over model parameters — temperature, top-k, max tokens, and system instructions. Test prompts and see responses instantly. This is the developer playground for tuning on-device AI behavior.

The $33B Opportunity

The on-device AI market is projected to hit $33 billion this year and $156 billion by 2033. Every mobile app will eventually need this. The question is whether you build the infrastructure yourself or use a toolkit that's already built. Google released their AI Edge Gallery (22.5k stars on GitHub) — but it's native Android/iOS only. There was nothing for Flutter's 1M+ developers. So I built it.

Open Source. Free. Apache 2.0.

EdgeAI Kit is fully open source. Use it in your apps, modify it, contribute to it. ⭐ github.com/sumitvairagar/edge-ai-kit If you're building mobile apps that handle sensitive data, or if you're tired of paying per-token API costs, take a look. PRs welcome.

What's Next

I'm working on:

Image and audio input — Gemma 4 supports multimodal, the toolkit will too
Conversation persistence — Save and resume on-device chats
Model fine-tuning hooks — Customize models for specific domains
A Flutter package — So you can drop on-device AI into any app with a single dependency

Building privacy-first AI solutions for mobile? Let's connect — I help teams integrate on-device AI into their Flutter apps. Reach out on LinkedIn.