Building

Computer-use for Android: Build the Control Layer Before the Agent

A practical development direction for building an Android agent around real capabilities, evidence, and verified workflows.

Computer-use for Android: Build the Control Layer Before the Agent

Sep 9, 2026 - 4 min read

  • Android
  • AI agents
  • Automation
  • Accessibility
  • Product development

I am exploring a native Android application that can act as a computer-use layer for a phone. The long-term goal is not just to put a chat box on top of Android. It is to build a system that can understand a task, choose from explicit capabilities, perform actions, observe the result, verify it, and continue safely.

The core loop is:

Understand -> Decide -> Act -> Observe -> Verify -> Continue

An example request might be: “Open my files, find the PDF I downloaded today, move it into my Physics folder, and tell me when it is done.” That sounds like one sentence, but it contains several separate problems: finding the right application, understanding the current screen, selecting an item, performing a file operation, checking that the move succeeded, and communicating the result.

Real capability over impressive wording

The most important rule for this project is simple: no fake progress.

Reading about Android does not mean the application can control Android. Writing code that has never run does not count as an implemented capability. A milestone counts when it produces an observable chain:

Input -> System -> Real result

The first meaningful result can be very small. If my application receives a command and successfully opens another application through an Android Intent, that is a real foundation. It is more valuable than claiming that an agent is “mostly built” because a prompt can generate a plan.

The development ladder

The project should grow through evidence-backed levels:

  1. Unknown: the concept is still unclear.
  2. Understood: I can explain it in my own words.
  3. Reproduced: I followed a working example and ran it.
  4. Modified: I changed the example and kept it working.
  5. Integrated: the capability exists inside my application.
  6. Reliable: it works repeatedly and handles common failures.
  7. Productized: it is part of a usable experience.

This keeps the project honest. A screenshot, test result, video, APK, or reproducible build should accompany important progress.

Start with Android itself

The first phase should focus on basic Android capabilities instead of an LLM. The application can begin with commands to open a browser, open Maps, and open Calculator. After that, it can build a deterministic Calculator workflow:

Open Calculator -> tap 2 -> tap 3 -> tap + -> tap 1 -> tap 9 -> tap = -> verify 42

This teaches the central idea behind an agent: an agent is not only a model. It is a system that observes state and takes actions toward a goal.

The next layer is an action engine with bounded operations such as openApp, tap, type, scroll, back, wait, and observe. The action engine should work manually before an AI model is allowed to choose actions.

Accessibility is an investigation, not a promise

Android’s AccessibilityService is an important area to investigate for UI automation. It may provide useful application and UI-node information, but it should not be treated as unrestricted control over the operating system.

The experiments should answer concrete questions:

  • Can the service detect the current application?
  • Can it inspect visible UI nodes?
  • Can it find an element?
  • Can it trigger an action?
  • Can it type into an appropriate field?
  • Can it scroll and navigate back?

Only after a small cross-app workflow works reliably should the project add an AI planner.

Add the AI brain later

When the deterministic control layer works, an AI model can produce a structured plan and select from explicit tools:

User request -> structured plan -> action engine -> Android -> observation -> next action

The model should not be given arbitrary access to Android operations. A constrained tool surface makes the behavior easier to inspect, test, and protect.

Screen capture, vision, voice input, background operation, and local inference are later layers. Android’s restrictions around accessibility, background services, permissions, and foreground-service types mean each capability needs a real device experiment instead of an architectural assumption.

The first definition of done

The project has started when this works:

Your phone -> your app -> “Open Calculator” -> Android -> Calculator opens

The first larger milestone is:

Your phone -> your app -> “Calculate 23 + 19” -> agent workflow -> Calculator -> result verified

That is enough to move the project from an idea to a real capability. Everything after that should extend the evidence chain one step at a time.

This direction is based on the Computer Use Agents planning conversation.