Google's Gemini Live Now Describes Your Surroundings Out Loud, and Tells You Where to Point the Camera
Point your camera at the world around you and Guided Vision narrates what it sees, including which way to move the lens. It works on Android 9 and later, and Google says it is not meant for navigation or obstacle detection.
On October 1 local time, Google launched Guided Vision, a visual assistance feature inside Gemini Live. Point your phone’s camera at your surroundings and the AI describes what appears on screen out loud, then answers follow-up questions. The feature is aimed at blind and low-vision users, as well as anyone in a situation where checking something visually is difficult.
It tells you where to move the camera
Guided Vision uses voice prompts to help you frame your subject better. If the camera is aimed too high or held too close, it will tell you to tilt down or step back. If the subject drifts off the edge of the frame, it guides you with instructions like moving slowly to the right.
The use cases Google showed off range from reading fine print to finding objects. You can use it to read nutrition information on food packaging, the cycle settings on a washing machine, or a printed menu in a dimly lit restaurant. It can also help you find earbuds dropped on the floor or the pepper in a cupboard crowded with seasonings, and confirm the color and pattern of a piece of clothing. You can also ask it to describe the layout of an unfamiliar room or the objects on a table.
More than 1,000 people tested it in daily life
Google said it worked with Aira to train Gemini Live to respond conversationally in real-world situations. The company said the work of interpreting visual information took tens of thousands of hours in total. More than 1,000 participants in Aira’s Trusted Tester network tried the model during everyday activities and contributed to its refinement. Aira’s experts also helped establish and evaluate safety standards.
Source: Google (press release)
Multilingual conversation is another feature. Google said it collected test results and feedback from blind and low-vision users in countries including India, Brazil, Singapore, Indonesia, and Japan. That feedback went into making the scene descriptions, camera framing guidance, and answers feel more natural. Google also named older adults, people who have difficulty reading text, and anyone who needs to read small print in low light as potential users.
How to turn it on
It is available on devices running Android 9 or later, in regions and languages where Gemini Live is supported. Turn on “Use Guided Vision in Live” in your profile settings in the Gemini app, then share your camera in Gemini Live. Google advised updating the Gemini app and Android system software to the latest versions before using it.
Source: Google (press release)
You can also launch it directly from accessibility features. Assign a Guided Vision shortcut in the accessibility menu in Android settings, and you can start it with the on-screen accessibility button, a two-finger swipe, or by pressing both volume buttons. Users of Google TalkBack, which reads screen content aloud, can tap the screen with three fingers to open the menu and select Guided Vision.
Google stressed that generative AI can make mistakes, and that Guided Vision is not a medical device or a mobility aid, nor a replacement for a white cane. The company was explicit that it is also not intended for navigation, safe-travel guidance, or obstacle detection. What remains to be seen is how accurately it describes fine print and objects, and how genuinely useful the camera framing guidance turns out to be in practice.