In Short:
Google Lens, introduced in 2017, lets users identify objects using their camera. Now, it will support more search methods, like video and voice commands. Lens will provide detailed shopping options, including direct links and reviews. An experimental feature allows users to capture real-time videos for assistance with repairs. These updates hint at a future where smart glasses might further enhance this experience.
Google Lens, launched in 2017, revolutionized the way users interact with the world around them through technology. This search feature allows individuals to point their phone’s camera at an object, enabling Google Lens to identify it, provide context, and even facilitate purchases. This innovation marked a significant shift in searching, eliminating the need for cumbersome text descriptions of visible items.
As Google integrates more machine learning and AI capabilities, the functionality of Lens continues to evolve. Currently, Lens handles approximately 20 billion searches each month, and the company is expanding its capabilities to include video and multimodal searches. This enhancement reflects Google’s commitment to optimizing its foundational generative AI models for both visual and text-based search responses.
Enhanced Shopping Experience
One of the notable updates to Lens is its ability to provide more contextual information for shopping-related queries. Visual search plays a critical role in purchasing decisions, and competitors like Amazon and Pinterest have similarly developed visual search tools. Previously, searching for a pair of sneakers via Lens resulted in a carousel of similar items. The updated Lens will feature direct links for purchases, customer and publisher reviews, and comparative shopping tools, enhancing the overall shopping experience.
Multimodal Search Capabilities
The latest upgrade introduces multimodal search capabilities, enabling users to combine video, images, and voice inputs for a more interactive experience. Rather than merely pointing their smartphone at an object and tapping the screen for results, users can now ask voice questions, such as “What kind of clouds are those?” or “What brand of sneakers are those and where can I buy them?” while using Lens.
Real-Time Video Capture
In an innovative development, Lens will also support real-time video capture, allowing users to receive assistance in identifying objects in motion. For instance, if a user encounters a malfunctioning appliance, they can record a brief video and receive AI-generated repair tips based on the footage.
Future Features and Experimental Options
Initially announced at the Google I/O event, this “video understanding” feature remains experimental and is currently accessible only to users who have opted into Google’s search labs, according to Rajan Patel, a co-founder of Lens and seasoned Google employee. Other features, such as voice mode and expanded shopping options, are being rolled out more widely.
The potential for this video understanding capability extends beyond real-time applications. When fully developed, it could allow users to tag and shop items within their own video collections or access vast databases like Google’s video library.
Project Astra and Future Innovations
This Lens feature bears similarities to Google’s Project Astra, which is set to launch later this year. Like Lens, Astra employs multimodal inputs to analyze and interpret surroundings through mobile devices. During a recent demo, Google showcased prototype smart glasses as part of the Astra initiative.
Meanwhile, Meta has unveiled its ambitious long-term vision for augmented reality, featuring glasses capable of intelligently interpreting the environment and presenting holographic interfaces. Although Google previously attempted to introduce a similar concept with Google Glass, which relied on different technology, the new features of Lens, in conjunction with Astra, may pave the way for the next generation of smart glasses.
