Reports: Google is working on “Screen Automation” for Gemini on Android

Google is developing an update for the Gemini assistant on Android that enables it to perform actions within third-party apps. 9to5Google discovered references to this “screen automation” feature in the Gemini app beta (version 17.4).

How Screen Automation works

The feature allows Gemini to interact with other apps to complete tasks. Instead of manual navigation, users can dictate commands for the assistant to execute within the relevant application. Code strings indicate that Google will warn users about potential errors, as the device owner remains responsible for any automated actions. At launch, the feature will likely support a limited number of apps.

UI redesign and new menus

Google is also redesigning Gemini’s floating interface. Quick action buttons, such as “Ask about this PDF” or “Ask about this place,” will move from the main screen to a new “Plus” menu. Additionally, the option to share conversations will become more prominent in the top-right corner of the chat interface.

3D avatars and system evolution

Google is developing 3D avatars for Gemini, similar to Apple’s “Persona” system used in the Vision Pro. This suggests a move toward more expressive, identity-driven AI interactions. Although unannounced, these beta features indicate that Gemini is evolving from a voice assistant into a system-level automation tool.

Source: 9to5Google